You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by rm...@apache.org on 2020/03/30 09:53:20 UTC
[flink-web] branch asf-site updated: regnerate to fix links on
index.html
This is an automated email from the ASF dual-hosted git repository.
rmetzger pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/flink-web.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 15ac3c8 regnerate to fix links on index.html
15ac3c8 is described below
commit 15ac3c80fecf14b97f8baa1f61d576157fccacad
Author: Robert Metzger <rm...@apache.org>
AuthorDate: Mon Mar 30 11:53:08 2020 +0200
regnerate to fix links on index.html
---
content/blog/feed.xml | 19314 +++++++++++++++++++++++++-----------------------
content/index.html | 2 +-
content/zh/index.html | 2 +-
3 files changed, 9988 insertions(+), 9330 deletions(-)
diff --git a/content/blog/feed.xml b/content/blog/feed.xml
index e204548..6d57b1b 100644
--- a/content/blog/feed.xml
+++ b/content/blog/feed.xml
@@ -8,2008 +8,2007 @@
<item>
<title>Flink as Unified Engine for Modern Data Warehousing: Production-Ready Hive Integration</title>
-<description>In this blog post, you will learn our motivation behind the Flink-Hive integration, and how Flink 1.10 can help modernize your data warehouse.
-
-{% toc %}
-
-
-## Introduction
-
-What are some of the latest requirements for your data warehouse and data infrastructure in 2020?
-
-We’ve came up with some for you.
-
-Firstly, today’s business is shifting to a more real-time fashion, and thus demands abilities to process online streaming data with low latency for near-real-time or even real-time analytics. People become less and less tolerant of delays between when data is generated and when it arrives at their hands, ready to use. Hours or even days of delay is not acceptable anymore. Users are expecting minutes, or even seconds, of end-to-end latency for data in their warehouse, to get quicker-than- [...]
-
-Secondly, the infrastructure should be able to handle both offline batch data for offline analytics and exploration, and online streaming data for more timely analytics. Both are indispensable as they both have very valid use cases. Apart from the real time processing mentioned above, batch processing would still exist as it’s good for ad hoc queries and explorations, and full-size calculations. Your modern infrastructure should not force users to choose between one or the other, it shou [...]
-
-Thirdly, the data players, including data engineers, data scientists, analysts, and operations, urge a more unified infrastructure than ever before for easier ramp-up and higher working efficiency. The big data landscape has been fragmented for years - companies may have one set of infrastructure for real time processing, one set for batch, one set for OLAP, etc. That, oftentimes, comes as a result of the legacy of lambda architecture, which was popular in the era when stream processors [...]
-
-If any of these resonate with you, you just found the right post to read: we have never been this close to the vision by strengthening Flink’s integration with Hive to a production grade.
+<description><p>In this blog post, you will learn our motivation behind the Flink-Hive integration, and how Flink 1.10 can help modernize your data warehouse.</p>
+
+<div class="page-toc">
+<ul id="markdown-toc">
+ <li><a href="#introduction" id="markdown-toc-introduction">Introduction</a></li>
+ <li><a href="#flink-and-its-integration-with-hive-comes-into-the-scene" id="markdown-toc-flink-and-its-integration-with-hive-comes-into-the-scene">Flink and Its Integration With Hive Comes into the Scene</a> <ul>
+ <li><a href="#unified-metadata-management" id="markdown-toc-unified-metadata-management">Unified Metadata Management</a></li>
+ <li><a href="#stream-processing" id="markdown-toc-stream-processing">Stream Processing</a></li>
+ <li><a href="#compatible-with-more-hive-versions" id="markdown-toc-compatible-with-more-hive-versions">Compatible with More Hive Versions</a></li>
+ <li><a href="#reuse-hive-user-defined-functions-udfs" id="markdown-toc-reuse-hive-user-defined-functions-udfs">Reuse Hive User Defined Functions (UDFs)</a></li>
+ <li><a href="#enhanced-read-and-write-on-hive-data" id="markdown-toc-enhanced-read-and-write-on-hive-data">Enhanced Read and Write on Hive Data</a></li>
+ <li><a href="#formats" id="markdown-toc-formats">Formats</a></li>
+ <li><a href="#more-data-types" id="markdown-toc-more-data-types">More Data Types</a></li>
+ <li><a href="#roadmap" id="markdown-toc-roadmap">Roadmap</a></li>
+ </ul>
+ </li>
+ <li><a href="#summary" id="markdown-toc-summary">Summary</a></li>
+</ul>
+</div>
-## Flink and Its Integration With Hive Comes into the Scene
+<h2 id="introduction">Introduction</h2>
-Apache Flink has been a proven scalable system to handle extremely high workload of streaming data in super low latency in many giant tech companies.
+<p>What are some of the latest requirements for your data warehouse and data infrastructure in 2020?</p>
-Despite its huge success in the real time processing domain, at its deep root, Flink has been faithfully following its inborn philosophy of being [a unified data processing engine for both batch and streaming](https://flink.apache.org/news/2019/02/13/unified-batch-streaming-blink.html), and taking a streaming-first approach in its architecture to do batch processing. By making batch a special case for streaming, Flink really leverages its cutting edge streaming capabilities and applies t [...]
+<p>We’ve came up with some for you.</p>
-On the other hand, Apache Hive has established itself as a focal point of the data warehousing ecosystem. It serves as not only a SQL engine for big data analytics and ETL, but also a data management platform, where data is discovered and defined. As business evolves, it puts new requirements on data warehouse.
+<p>Firstly, today’s business is shifting to a more real-time fashion, and thus demands abilities to process online streaming data with low latency for near-real-time or even real-time analytics. People become less and less tolerant of delays between when data is generated and when it arrives at their hands, ready to use. Hours or even days of delay is not acceptable anymore. Users are expecting minutes, or even seconds, of end-to-end latency for data in their warehouse, to get quic [...]
-Thus we started integrating Flink and Hive as a beta version in Flink 1.9. Over the past few months, we have been listening to users’ requests and feedback, extensively enhancing our product, and running rigorous benchmarks (which will be published soon separately). I’m glad to announce that the integration between Flink and Hive is at production grade in [Flink 1.10](https://flink.apache.org/news/2020/02/11/release-1.10.0.html) and we can’t wait to walk you through the details.
+<p>Secondly, the infrastructure should be able to handle both offline batch data for offline analytics and exploration, and online streaming data for more timely analytics. Both are indispensable as they both have very valid use cases. Apart from the real time processing mentioned above, batch processing would still exist as it’s good for ad hoc queries and explorations, and full-size calculations. Your modern infrastructure should not force users to choose between one or the other [...]
+<p>Thirdly, the data players, including data engineers, data scientists, analysts, and operations, urge a more unified infrastructure than ever before for easier ramp-up and higher working efficiency. The big data landscape has been fragmented for years - companies may have one set of infrastructure for real time processing, one set for batch, one set for OLAP, etc. That, oftentimes, comes as a result of the legacy of lambda architecture, which was popular in the era when stream pr [...]
-### Unified Metadata Management
+<p>If any of these resonate with you, you just found the right post to read: we have never been this close to the vision by strengthening Flink’s integration with Hive to a production grade.</p>
-Hive Metastore has evolved into the de facto metadata hub over the years in the Hadoop, or even the cloud, ecosystem. Many companies have a single Hive Metastore service instance in production to manage all of their schemas, either Hive or non-Hive metadata, as the single source of truth.
+<h2 id="flink-and-its-integration-with-hive-comes-into-the-scene">Flink and Its Integration With Hive Comes into the Scene</h2>
-In 1.9 we introduced Flink’s [HiveCatalog](https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/hive/hive_catalog.html), connecting Flink to users’ rich metadata pool. The meaning of `HiveCatalog` is two-fold here. First, it allows Apache Flink users to utilize Hive Metastore to store and manage Flink’s metadata, including tables, UDFs, and statistics of data. Second, it enables Flink to access Hive’s existing metadata, so that Flink itself can read and write Hive tables.
+<p>Apache Flink has been a proven scalable system to handle extremely high workload of streaming data in super low latency in many giant tech companies.</p>
-In Flink 1.10, users can store Flink's own tables, views, UDFs, statistics in Hive Metastore on all of the compatible Hive versions mentioned above. [Here’s an end-to-end example](https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/hive/hive_catalog.html#example) of how to store a Flink’s Kafka source table in Hive Metastore and later query the table in Flink SQL.
+<p>Despite its huge success in the real time processing domain, at its deep root, Flink has been faithfully following its inborn philosophy of being <a href="https://flink.apache.org/news/2019/02/13/unified-batch-streaming-blink.html">a unified data processing engine for both batch and streaming</a>, and taking a streaming-first approach in its architecture to do batch processing. By making batch a special case for streaming, Flink really leverages its cutting [...]
+<p>On the other hand, Apache Hive has established itself as a focal point of the data warehousing ecosystem. It serves as not only a SQL engine for big data analytics and ETL, but also a data management platform, where data is discovered and defined. As business evolves, it puts new requirements on data warehouse.</p>
-### Stream Processing
+<p>Thus we started integrating Flink and Hive as a beta version in Flink 1.9. Over the past few months, we have been listening to users’ requests and feedback, extensively enhancing our product, and running rigorous benchmarks (which will be published soon separately). I’m glad to announce that the integration between Flink and Hive is at production grade in <a href="https://flink.apache.org/news/2020/02/11/release-1.10.0.html">Flink 1.10</a> and we can’t wait [...]
-The Hive integration feature in Flink 1.10 empowers users to re-imagine what they can accomplish with their Hive data and unlock stream processing use cases:
+<h3 id="unified-metadata-management">Unified Metadata Management</h3>
-- join real-time streaming data in Flink with offline Hive data for more complex data processing
-- backfill Hive data with Flink directly in a unified fashion
-- leverage Flink to move real-time data into Hive more quickly, greatly shortening the end-to-end latency between when data is generated and when it arrives at your data warehouse for analytics, from hours — or even days — to minutes
+<p>Hive Metastore has evolved into the de facto metadata hub over the years in the Hadoop, or even the cloud, ecosystem. Many companies have a single Hive Metastore service instance in production to manage all of their schemas, either Hive or non-Hive metadata, as the single source of truth.</p>
+<p>In 1.9 we introduced Flink’s <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/hive/hive_catalog.html">HiveCatalog</a>, connecting Flink to users’ rich metadata pool. The meaning of <code>HiveCatalog</code> is two-fold here. First, it allows Apache Flink users to utilize Hive Metastore to store and manage Flink’s metadata, including tables, UDFs, and statistics of data. Second, it enables Flink to access Hive’s exis [...]
-### Compatible with More Hive Versions
+<p>In Flink 1.10, users can store Flink’s own tables, views, UDFs, statistics in Hive Metastore on all of the compatible Hive versions mentioned above. <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/hive/hive_catalog.html#example">Here’s an end-to-end example</a> of how to store a Flink’s Kafka source table in Hive Metastore and later query the table in Flink SQL.</p>
-In Flink 1.10, we brought full coverage to most Hive versions including 1.0, 1.1, 1.2, 2.0, 2.1, 2.2, 2.3, and 3.1. Take a look [here](https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/hive/#supported-hive-versions).
+<h3 id="stream-processing">Stream Processing</h3>
+<p>The Hive integration feature in Flink 1.10 empowers users to re-imagine what they can accomplish with their Hive data and unlock stream processing use cases:</p>
-### Reuse Hive User Defined Functions (UDFs)
+<ul>
+ <li>join real-time streaming data in Flink with offline Hive data for more complex data processing</li>
+ <li>backfill Hive data with Flink directly in a unified fashion</li>
+ <li>leverage Flink to move real-time data into Hive more quickly, greatly shortening the end-to-end latency between when data is generated and when it arrives at your data warehouse for analytics, from hours — or even days — to minutes</li>
+</ul>
-Users can [reuse all kinds of Hive UDFs in Flink](https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/hive/hive_functions.html#hive-user-defined-functions) since Flink 1.9.
+<h3 id="compatible-with-more-hive-versions">Compatible with More Hive Versions</h3>
-This is a great win for Flink users with past history with the Hive ecosystem, as they may have developed custom business logic in their Hive UDFs. Being able to run these functions without any rewrite saves users a lot of time and brings them a much smoother experience when they migrate to Flink.
+<p>In Flink 1.10, we brought full coverage to most Hive versions including 1.0, 1.1, 1.2, 2.0, 2.1, 2.2, 2.3, and 3.1. Take a look <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/hive/#supported-hive-versions">here</a>.</p>
-To take it a step further, Flink 1.10 introduces [compatibility of Hive built-in functions via HiveModule](https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/hive/hive_functions.html#use-hive-built-in-functions-via-hivemodule). Over the years, the Hive community has developed a few hundreds of built-in functions that are super handy for users. For those built-in functions that don't exist in Flink yet, users are now able to leverage the existing Hive built-in func [...]
+<h3 id="reuse-hive-user-defined-functions-udfs">Reuse Hive User Defined Functions (UDFs)</h3>
+<p>Users can <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/hive/hive_functions.html#hive-user-defined-functions">reuse all kinds of Hive UDFs in Flink</a> since Flink 1.9.</p>
-### Enhanced Read and Write on Hive Data
+<p>This is a great win for Flink users with past history with the Hive ecosystem, as they may have developed custom business logic in their Hive UDFs. Being able to run these functions without any rewrite saves users a lot of time and brings them a much smoother experience when they migrate to Flink.</p>
-Flink 1.10 extends its read and write capabilities on Hive data to all the common use cases with better performance.
+<p>To take it a step further, Flink 1.10 introduces <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/hive/hive_functions.html#use-hive-built-in-functions-via-hivemodule">compatibility of Hive built-in functions via HiveModule</a>. Over the years, the Hive community has developed a few hundreds of built-in functions that are super handy for users. For those built-in functions that don’t exist in Flink yet, users are now able to le [...]
-On the reading side, Flink now can read Hive regular tables, partitioned tables, and views. Lots of optimization techniques are developed around reading, including partition pruning and projection pushdown to transport less data from file storage, limit pushdown for faster experiment and exploration, and vectorized reader for ORC files.
+<h3 id="enhanced-read-and-write-on-hive-data">Enhanced Read and Write on Hive Data</h3>
-On the writing side, Flink 1.10 introduces “INSERT INTO” and “INSERT OVERWRITE” to its syntax, and can write to not only Hive’s regular tables, but also partitioned tables with either static or dynamic partitions.
+<p>Flink 1.10 extends its read and write capabilities on Hive data to all the common use cases with better performance.</p>
-### Formats
+<p>On the reading side, Flink now can read Hive regular tables, partitioned tables, and views. Lots of optimization techniques are developed around reading, including partition pruning and projection pushdown to transport less data from file storage, limit pushdown for faster experiment and exploration, and vectorized reader for ORC files.</p>
-Your engine should be able to handle all common types of file formats to give you the freedom of choosing one over another in order to fit your business needs. It’s no exception for Flink. We have tested the following table storage formats: text, csv, SequenceFile, ORC, and Parquet.
+<p>On the writing side, Flink 1.10 introduces “INSERT INTO” and “INSERT OVERWRITE” to its syntax, and can write to not only Hive’s regular tables, but also partitioned tables with either static or dynamic partitions.</p>
-### More Data Types
+<h3 id="formats">Formats</h3>
-In Flink 1.10, we added support for a few more frequently-used Hive data types that were not covered by Flink 1.9. Flink users now should have a full, smooth experience to query and manipulate Hive data from Flink.
+<p>Your engine should be able to handle all common types of file formats to give you the freedom of choosing one over another in order to fit your business needs. It’s no exception for Flink. We have tested the following table storage formats: text, csv, SequenceFile, ORC, and Parquet.</p>
+<h3 id="more-data-types">More Data Types</h3>
-### Roadmap
+<p>In Flink 1.10, we added support for a few more frequently-used Hive data types that were not covered by Flink 1.9. Flink users now should have a full, smooth experience to query and manipulate Hive data from Flink.</p>
-Integration between any two systems is a never-ending story.
+<h3 id="roadmap">Roadmap</h3>
-We are constantly improving Flink itself and the Flink-Hive integration also gets improved by collecting user feedback and working with folks in this vibrant community.
+<p>Integration between any two systems is a never-ending story.</p>
-After careful consideration and prioritization of the feedback we received, we have prioritize many of the below requests for the next Flink release of 1.11.
+<p>We are constantly improving Flink itself and the Flink-Hive integration also gets improved by collecting user feedback and working with folks in this vibrant community.</p>
-- Hive streaming sink so that Flink can stream data into Hive tables, bringing a real streaming experience to Hive
-- Native Parquet reader for better performance
-- Additional interoperability - support creating Hive tables, views, functions in Flink
-- Better out-of-box experience with built-in dependencies, including documentations
-- JDBC driver so that users can reuse their existing toolings to run SQL jobs on Flink
-- Hive syntax and semantic compatible mode
+<p>After careful consideration and prioritization of the feedback we received, we have prioritize many of the below requests for the next Flink release of 1.11.</p>
-If you have more feature requests or discover bugs, please reach out to the community through mailing list and JIRAs.
+<ul>
+ <li>Hive streaming sink so that Flink can stream data into Hive tables, bringing a real streaming experience to Hive</li>
+ <li>Native Parquet reader for better performance</li>
+ <li>Additional interoperability - support creating Hive tables, views, functions in Flink</li>
+ <li>Better out-of-box experience with built-in dependencies, including documentations</li>
+ <li>JDBC driver so that users can reuse their existing toolings to run SQL jobs on Flink</li>
+ <li>Hive syntax and semantic compatible mode</li>
+</ul>
+<p>If you have more feature requests or discover bugs, please reach out to the community through mailing list and JIRAs.</p>
-## Summary
+<h2 id="summary">Summary</h2>
-Data warehousing is shifting to a more real-time fashion, and Apache Flink can make a difference for your organization in this space.
+<p>Data warehousing is shifting to a more real-time fashion, and Apache Flink can make a difference for your organization in this space.</p>
-Flink 1.10 brings production-ready Hive integration and empowers users to achieve more in both metadata management and unified/batch data processing.
+<p>Flink 1.10 brings production-ready Hive integration and empowers users to achieve more in both metadata management and unified/batch data processing.</p>
-We encourage all our users to get their hands on Flink 1.10. You are very welcome to join the community in development, discussions, and all other kinds of collaborations in this topic.
+<p>We encourage all our users to get their hands on Flink 1.10. You are very welcome to join the community in development, discussions, and all other kinds of collaborations in this topic.</p>
</description>
-<pubDate>Fri, 27 Mar 2020 02:30:00 +0000</pubDate>
+<pubDate>Fri, 27 Mar 2020 03:30:00 +0100</pubDate>
<link>https://flink.apache.org/features/2020/03/27/flink-for-data-warehouse.html</link>
<guid isPermaLink="true">/features/2020/03/27/flink-for-data-warehouse.html</guid>
</item>
<item>
<title>Advanced Flink Application Patterns Vol.2: Dynamic Updates of Application Logic</title>
-<description>In the [first article](https://flink.apache.org/news/2020/01/15/demo-fraud-detection.html) of the series, we gave a high-level description of the objectives and required functionality of a Fraud Detection engine. We also described how to make data partitioning in Apache Flink customizable based on modifiable rules instead of using a hardcoded `KeysExtractor` implementation.
+<description><p>In the <a href="https://flink.apache.org/news/2020/01/15/demo-fraud-detection.html">first article</a> of the series, we gave a high-level description of the objectives and required functionality of a Fraud Detection engine. We also described how to make data partitioning in Apache Flink customizable based on modifiable rules instead of using a hardcoded <code>KeysExtractor</code> implementation.</p>
-We intentionally omitted details of how the applied rules are initialized and what possibilities exist for updating them at runtime. In this post, we will address exactly these details. You will learn how the approach to data partitioning described in [Part 1](https://flink.apache.org/news/2020/01/15/demo-fraud-detection.html) can be applied in combination with a dynamic configuration. These two patterns, when used together, can eliminate the need to recompile the code and redeploy your [...]
+<p>We intentionally omitted details of how the applied rules are initialized and what possibilities exist for updating them at runtime. In this post, we will address exactly these details. You will learn how the approach to data partitioning described in <a href="https://flink.apache.org/news/2020/01/15/demo-fraud-detection.html">Part 1</a> can be applied in combination with a dynamic configuration. These two patterns, when used together, can eliminate the nee [...]
-## Rules Broadcasting
+<h2 id="rules-broadcasting">Rules Broadcasting</h2>
-Let's first have a look at the [previously-defined](https://flink.apache.org/news/2020/01/15/demo-fraud-detection.html#dynamic-data-partitioning) data-processing pipeline:
+<p>Let’s first have a look at the <a href="https://flink.apache.org/news/2020/01/15/demo-fraud-detection.html#dynamic-data-partitioning">previously-defined</a> data-processing pipeline:</p>
-```java
-DataStream<Alert> alerts =
- transactions
- .process(new DynamicKeyFunction())
- .keyBy((keyed) -> keyed.getKey());
- .process(new DynamicAlertFunction())
-```
+<div class="highlight"><pre><code class="language-java"><span class="n">DataStream</span><span class="o">&lt;</span><span class="n">Alert</span><span class="o">&gt;</span> <span class="n">alerts</span> <span class="o">=</span>
+ <span class="n">transactions</span>
+ <span class="o">.</span><span class="na">process</span><span class="o">(</span><span class="k">new</span> <span class="nf">DynamicKeyFunction</span><span class="o">())</span>
+ <span class="o">.</span><span class="na">keyBy</span><span class="o">((</span><span class="n">keyed</span><span class="o">)</span> <span class="o">-&gt;</span> <span class="n">keyed</span><span class="o">.</span><span class="na">getKey</span><span class="o">());< [...]
+ <span class="o">.</span><span class="na">process</span><span class="o">(</span><span class="k">new</span> <span class="nf">DynamicAlertFunction</span><span class="o">())</span></code></pre></div>
-`DynamicKeyFunction` provides dynamic data partitioning while `DynamicAlertFunction` is responsible for executing the main logic of processing transactions and sending alert messages according to defined rules.
+<p><code>DynamicKeyFunction</code> provides dynamic data partitioning while <code>DynamicAlertFunction</code> is responsible for executing the main logic of processing transactions and sending alert messages according to defined rules.</p>
-Vol.1 of this series simplified the use case and assumed that the applied set of rules is pre-initialized and accessible via the `List<Rules>` within `DynamicKeyFunction`.
+<p>Vol.1 of this series simplified the use case and assumed that the applied set of rules is pre-initialized and accessible via the <code>List&lt;Rules&gt;</code> within <code>DynamicKeyFunction</code>.</p>
-```java
-public class DynamicKeyFunction
- extends ProcessFunction<Transaction, Keyed<Transaction, String, Integer>> {
+<div class="highlight"><pre><code class="language-java"><span class="kd">public</span> <span class="kd">class</span> <span class="nc">DynamicKeyFunction</span>
+ <span class="kd">extends</span> <span class="n">ProcessFunction</span><span class="o">&lt;</span><span class="n">Transaction</span><span class="o">,</span> <span class="n">Keyed</span><span class="o">&lt;</span><span class="n">Transaction</span><span class="o">,</span> <span cla [...]
- /* Simplified */
- List<Rule> rules = /* Rules that are initialized somehow.*/;
- ...
-}
-```
+ <span class="cm">/* Simplified */</span>
+ <span class="n">List</span><span class="o">&lt;</span><span class="n">Rule</span><span class="o">&gt;</span> <span class="n">rules</span> <span class="o">=</span> <span class="cm">/* Rules that are initialized somehow.*/</span><span class="o">;</span>
+ <span class="o">...</span>
+<span class="o">}</span></code></pre></div>
-Adding rules to this list is obviously possible directly inside the code of the Flink Job at the stage of its initialization (Create a `List` object; use it's `add` method). A major drawback of doing so is that it will require recompilation of the job with each rule modification. In a real Fraud Detection system, rules are expected to change on a frequent basis, making this approach unacceptable from the point of view of business and operational requirements. A different approach is needed.
+<p>Adding rules to this list is obviously possible directly inside the code of the Flink Job at the stage of its initialization (Create a <code>List</code> object; use it’s <code>add</code> method). A major drawback of doing so is that it will require recompilation of the job with each rule modification. In a real Fraud Detection system, rules are expected to change on a frequent basis, making this approach unacceptable from the point of view of business and [...]
-Next, let's take a look at a sample rule definition that we introduced in the previous post of the series:
+<p>Next, let’s take a look at a sample rule definition that we introduced in the previous post of the series:</p>
<center>
-<img src="{{ site.baseurl }}/img/blog/patterns-blog-2/rule-dsl.png" width="800px" alt="Figure 1: Rule definition"/>
-<br/>
+<img src="/img/blog/patterns-blog-2/rule-dsl.png" width="800px" alt="Figure 1: Rule definition" />
+<br />
<i><small>Figure 1: Rule definition</small></i>
</center>
-<br/>
+<p><br /></p>
-The previous post covered use of `groupingKeyNames` by `DynamicKeyFunction` to extract message keys. Parameters from the second part of this rule are used by `DynamicAlertFunction`: they define the actual logic of the performed operations and their parameters (such as the alert-triggering limit). This means that the same rule must be present in both `DynamicKeyFunction` and `DynamicAlertFunction`. To achieve this result, we will use the [broadcast data distribution mechanism](https://ci. [...]
+<p>The previous post covered use of <code>groupingKeyNames</code> by <code>DynamicKeyFunction</code> to extract message keys. Parameters from the second part of this rule are used by <code>DynamicAlertFunction</code>: they define the actual logic of the performed operations and their parameters (such as the alert-triggering limit). This means that the same rule must be present in both <code>DynamicKeyFunction</code> and <code>Dy [...]
-Figure 2 presents the final job graph of the system that we are building:
+<p>Figure 2 presents the final job graph of the system that we are building:</p>
<center>
-<img src="{{ site.baseurl }}/img/blog/patterns-blog-2/job-graph.png" width="800px" alt="Figure 2: Job Graph of the Fraud Detection Flink Job"/>
-<br/>
+<img src="/img/blog/patterns-blog-2/job-graph.png" width="800px" alt="Figure 2: Job Graph of the Fraud Detection Flink Job" />
+<br />
<i><small>Figure 2: Job Graph of the Fraud Detection Flink Job</small></i>
</center>
-<br/>
-
-The main blocks of the Transactions processing pipeline are:<br>
+<p><br /></p>
-* **Transaction Source** that consumes transaction messages from Kafka partitions in parallel. <br>
+<p>The main blocks of the Transactions processing pipeline are:<br /></p>
-* **Dynamic Key Function** that performs data enrichment with a dynamic key. The subsequent `keyBy` hashes this dynamic key and partitions the data accordingly among all parallel instances of the following operator.
-
-* **Dynamic Alert Function** that accumulates a data window and creates Alerts based on it.
+<ul>
+ <li>
+ <p><strong>Transaction Source</strong> that consumes transaction messages from Kafka partitions in parallel. <br /></p>
+ </li>
+ <li>
+ <p><strong>Dynamic Key Function</strong> that performs data enrichment with a dynamic key. The subsequent <code>keyBy</code> hashes this dynamic key and partitions the data accordingly among all parallel instances of the following operator.</p>
+ </li>
+ <li>
+ <p><strong>Dynamic Alert Function</strong> that accumulates a data window and creates Alerts based on it.</p>
+ </li>
+</ul>
-## Data Exchange inside Apache Flink
+<h2 id="data-exchange-inside-apache-flink">Data Exchange inside Apache Flink</h2>
-The job graph above also indicates various data exchange patterns between the operators. In order to understand how the broadcast pattern works, let's take a short detour and discuss what methods of message propagation exist in Apache Flink's distributed runtime.
+<p>The job graph above also indicates various data exchange patterns between the operators. In order to understand how the broadcast pattern works, let’s take a short detour and discuss what methods of message propagation exist in Apache Flink’s distributed runtime.</p>
-* The __FORWARD__ connection after the Transaction Source means that all data consumed by one of the parallel instances of the Transaction Source operator is transferred to exactly one instance of the subsequent `DynamicKeyFunction` operator. It also indicates the same level of parallelism of the two connected operators (12 in the above case). This communication pattern is illustrated in Figure 3. Orange circles represent transactions, and dotted rectangles depict parallel instances of t [...]
+<ul>
+ <li>The <strong>FORWARD</strong> connection after the Transaction Source means that all data consumed by one of the parallel instances of the Transaction Source operator is transferred to exactly one instance of the subsequent <code>DynamicKeyFunction</code> operator. It also indicates the same level of parallelism of the two connected operators (12 in the above case). This communication pattern is illustrated in Figure 3. Orange circles represent transact [...]
+</ul>
<center>
-<img src="{{ site.baseurl }}/img/blog/patterns-blog-2/forward.png" width="800px" alt="Figure 3: FORWARD message passing across operator instances"/>
-<br/>
+<img src="/img/blog/patterns-blog-2/forward.png" width="800px" alt="Figure 3: FORWARD message passing across operator instances" />
+<br />
<i><small>Figure 3: FORWARD message passing across operator instances</small></i>
</center>
-<br/>
+<p><br /></p>
-* The __HASH__ connection between `DynamicKeyFunction` and `DynamicAlertFunction` means that for each message a hash code is calculated and messages are evenly distributed among available parallel instances of the next operator. Such a connection needs to be explicitly "requested" from Flink by using `keyBy`.
+<ul>
+ <li>The <strong>HASH</strong> connection between <code>DynamicKeyFunction</code> and <code>DynamicAlertFunction</code> means that for each message a hash code is calculated and messages are evenly distributed among available parallel instances of the next operator. Such a connection needs to be explicitly “requested” from Flink by using <code>keyBy</code>.</li>
+</ul>
<center>
-<img src="{{ site.baseurl }}/img/blog/patterns-blog-2/hash.png" width="800px" alt="Figure 4: HASHED message passing across operator instances (via `keyBy`)"/>
-<br/>
+<img src="/img/blog/patterns-blog-2/hash.png" width="800px" alt="Figure 4: HASHED message passing across operator instances (via `keyBy`)" />
+<br />
<i><small>Figure 4: HASHED message passing across operator instances (via `keyBy`)</small></i>
</center>
-<br/>
+<p><br /></p>
-* A __REBALANCE__ distribution is either caused by an explicit call to `rebalance()` or by a change of parallelism (12 -> 1 in the case of the job graph from Figure 2). Calling `rebalance()` causes data to be repartitioned in a round-robin fashion and can help to mitigate data skew in certain scenarios.
+<ul>
+ <li>A <strong>REBALANCE</strong> distribution is either caused by an explicit call to <code>rebalance()</code> or by a change of parallelism (12 -&gt; 1 in the case of the job graph from Figure 2). Calling <code>rebalance()</code> causes data to be repartitioned in a round-robin fashion and can help to mitigate data skew in certain scenarios.</li>
+</ul>
<center>
-<img src="{{ site.baseurl }}/img/blog/patterns-blog-2/rebalance.png" width="800px" alt="Figure 5: REBALANCE message passing across operator instances"/>
-<br/>
+<img src="/img/blog/patterns-blog-2/rebalance.png" width="800px" alt="Figure 5: REBALANCE message passing across operator instances" />
+<br />
<i><small>Figure 5: REBALANCE message passing across operator instances</small></i>
</center>
-<br/>
+<p><br /></p>
-The Fraud Detection job graph in Figure 2 contains an additional data source: _Rules Source_. It also consumes from Kafka. Rules are "mixed into" the main processing data flow through the __BROADCAST__ channel. Unlike other methods of transmitting data between operators, such as `forward`, `hash` or `rebalance` that make each message available for processing in only one of the parallel instances of the receiving operator, `broadcast` makes each message available at the input of [...]
+<p>The Fraud Detection job graph in Figure 2 contains an additional data source: <em>Rules Source</em>. It also consumes from Kafka. Rules are “mixed into” the main processing data flow through the <strong>BROADCAST</strong> channel. Unlike other methods of transmitting data between operators, such as <code>forward</code>, <code>hash</code> or <code>rebalance</code> that make each message available for processing in only o [...]
<center>
- <img src="{{ site.baseurl }}/img/blog/patterns-blog-2/broadcast.png" width="800px" alt="Figure 6: BROADCAST message passing across operator instances"/>
- <br/>
+ <img src="/img/blog/patterns-blog-2/broadcast.png" width="800px" alt="Figure 6: BROADCAST message passing across operator instances" />
+ <br />
<i><small>Figure 6: BROADCAST message passing across operator instances</small></i>
</center>
- <br/>
+<p><br /></p>
-<div class="alert alert-info" markdown="1">
-<span class="label label-info" style="display: inline-block"><span class="glyphicon glyphicon-info-sign" aria-hidden="true"></span> Note</span>
-There are actually a few more specialized data partitioning schemes in Flink which we did not mention here. If you want to find out more, please refer to Flink's documentation on __[stream partitioning](https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/#physical-partitioning)__.
+<div class="alert alert-info">
+ <p><span class="label label-info" style="display: inline-block"><span class="glyphicon glyphicon-info-sign" aria-hidden="true"></span> Note</span>
+There are actually a few more specialized data partitioning schemes in Flink which we did not mention here. If you want to find out more, please refer to Flink’s documentation on <strong><a href="https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/#physical-partitioning">stream partitioning</a></strong>.</p>
</div>
-## Broadcast State Pattern
+<h2 id="broadcast-state-pattern">Broadcast State Pattern</h2>
-In order to make use of the Rules Source, we need to "connect" it to the main data stream:
+<p>In order to make use of the Rules Source, we need to “connect” it to the main data stream:</p>
-```java
-// Streams setup
-DataStream<Transaction> transactions = [...]
-DataStream<Rule> rulesUpdateStream = [...]
+<div class="highlight"><pre><code class="language-java"><span class="c1">// Streams setup</span>
+<span class="n">DataStream</span><span class="o">&lt;</span><span class="n">Transaction</span><span class="o">&gt;</span> <span class="n">transactions</span> <span class="o">=</span> <span class="o">[...]</span>
+<span class="n">DataStream</span><span class="o">&lt;</span><span class="n">Rule</span><span class="o">&gt;</span> <span class="n">rulesUpdateStream</span> <span class="o">=</span> <span class="o">[...]</span>
-BroadcastStream<Rule> rulesStream = rulesUpdateStream.broadcast(RULES_STATE_DESCRIPTOR);
+<span class="n">BroadcastStream</span><span class="o">&lt;</span><span class="n">Rule</span><span class="o">&gt;</span> <span class="n">rulesStream</span> <span class="o">=</span> <span class="n">rulesUpdateStream</span><span class="o">.</span><span class="na">broadcast</span><span [...]
-// Processing pipeline setup
- DataStream<Alert> alerts =
- transactions
- .connect(rulesStream)
- .process(new DynamicKeyFunction())
- .keyBy((keyed) -> keyed.getKey())
- .connect(rulesStream)
- .process(new DynamicAlertFunction())
-```
+<span class="c1">// Processing pipeline setup</span>
+ <span class="n">DataStream</span><span class="o">&lt;</span><span class="n">Alert</span><span class="o">&gt;</span> <span class="n">alerts</span> <span class="o">=</span>
+ <span class="n">transactions</span>
+ <span class="o">.</span><span class="na">connect</span><span class="o">(</span><span class="n">rulesStream</span><span class="o">)</span>
+ <span class="o">.</span><span class="na">process</span><span class="o">(</span><span class="k">new</span> <span class="nf">DynamicKeyFunction</span><span class="o">())</span>
+ <span class="o">.</span><span class="na">keyBy</span><span class="o">((</span><span class="n">keyed</span><span class="o">)</span> <span class="o">-&gt;</span> <span class="n">keyed</span><span class="o">.</span><span class="na">getKey</span><span class="o">())< [...]
+ <span class="o">.</span><span class="na">connect</span><span class="o">(</span><span class="n">rulesStream</span><span class="o">)</span>
+ <span class="o">.</span><span class="na">process</span><span class="o">(</span><span class="k">new</span> <span class="nf">DynamicAlertFunction</span><span class="o">())</span></code></pre></div>
-As you can see, the broadcast stream can be created from any regular stream by calling the `broadcast` method and specifying a state descriptor. Flink assumes that broadcasted data needs to be stored and retrieved while processing events of the main data flow and, therefore, always automatically creates a corresponding _broadcast state_ from this state descriptor. This is different from any other Apache Flink state type in which you need to initialize it in the `open()` method of the pr [...]
+<p>As you can see, the broadcast stream can be created from any regular stream by calling the <code>broadcast</code> method and specifying a state descriptor. Flink assumes that broadcasted data needs to be stored and retrieved while processing events of the main data flow and, therefore, always automatically creates a corresponding <em>broadcast state</em> from this state descriptor. This is different from any other Apache Flink state type in which you need [...]
-```java
-public static final MapStateDescriptor<Integer, Rule> RULES_STATE_DESCRIPTOR =
- new MapStateDescriptor<>("rules", Integer.class, Rule.class);
-```
+<div class="highlight"><pre><code class="language-java"><span class="kd">public</span> <span class="kd">static</span> <span class="kd">final</span> <span class="n">MapStateDescriptor</span><span class="o">&lt;</span><span class="n">Integer</span><span class="o">,</span> <span class="n&qu [...]
+ <span class="k">new</span> <span class="n">MapStateDescriptor</span><span class="o">&lt;&gt;(</span><span class="s">&quot;rules&quot;</span><span class="o">,</span> <span class="n">Integer</span><span class="o">.</span><span class="na">class</span><span class="o">,</span&g [...]
-Connecting to `rulesStream` causes some changes in the signature of the processing functions. The previous article presented it in a slightly simplified way as a `ProcessFunction`. However, `DynamicKeyFunction` is actually a `BroadcastProcessFunction`.
+<p>Connecting to <code>rulesStream</code> causes some changes in the signature of the processing functions. The previous article presented it in a slightly simplified way as a <code>ProcessFunction</code>. However, <code>DynamicKeyFunction</code> is actually a <code>BroadcastProcessFunction</code>.</p>
-```java
-public abstract class BroadcastProcessFunction<IN1, IN2, OUT> {
+<div class="highlight"><pre><code class="language-java"><span class="kd">public</span> <span class="kd">abstract</span> <span class="kd">class</span> <span class="nc">BroadcastProcessFunction</span><span class="o">&lt;</span><span class="n">IN1</span><span class="o">,</span> <span class=" [...]
- public abstract void processElement(IN1 value,
- ReadOnlyContext ctx,
- Collector<OUT> out) throws Exception;
+ <span class="kd">public</span> <span class="kd">abstract</span> <span class="kt">void</span> <span class="nf">processElement</span><span class="o">(</span><span class="n">IN1</span> <span class="n">value</span><span class="o">,</span>
+ <span class="n">ReadOnlyContext</span> <span class="n">ctx</span><span class="o">,</span>
+ <span class="n">Collector</span><span class="o">&lt;</span><span class="n">OUT</span><span class="o">&gt;</span> <span class="n">out</span><span class="o">)</span> <span class="kd">throws</span> <span class="n">Exception</span><span class="o">;</span>
- public abstract void processBroadcastElement(IN2 value,
- Context ctx,
- Collector<OUT> out) throws Exception;
+ <span class="kd">public</span> <span class="kd">abstract</span> <span class="kt">void</span> <span class="nf">processBroadcastElement</span><span class="o">(</span><span class="n">IN2</span> <span class="n">value</span><span class="o">,</span>
+ <span class="n">Context</span> <span class="n">ctx</span><span class="o">,</span>
+ <span class="n">Collector</span><span class="o">&lt;</span><span class="n">OUT</span><span class="o">&gt;</span> <span class="n">out</span><span class="o">)</span> <span class="kd">throws</span> <span class="n">Exception</span><span class="o"> [...]
-}
-```
+<span class="o">}</span></code></pre></div>
-The difference is the addition of the `processBroadcastElement` method through which messages of the rules stream will arrive. The following new version of `DynamicKeyFunction` allows modifying the list of data-distribution keys at runtime through this stream:
+<p>The difference is the addition of the <code>processBroadcastElement</code> method through which messages of the rules stream will arrive. The following new version of <code>DynamicKeyFunction</code> allows modifying the list of data-distribution keys at runtime through this stream:</p>
-```java
-public class DynamicKeyFunction
- extends BroadcastProcessFunction<Transaction, Rule, Keyed<Transaction, String, Integer>> {
+<div class="highlight"><pre><code class="language-java"><span class="kd">public</span> <span class="kd">class</span> <span class="nc">DynamicKeyFunction</span>
+ <span class="kd">extends</span> <span class="n">BroadcastProcessFunction</span><span class="o">&lt;</span><span class="n">Transaction</span><span class="o">,</span> <span class="n">Rule</span><span class="o">,</span> <span class="n">Keyed</span><span class="o">&lt;</span><span c [...]
- @Override
- public void processBroadcastElement(Rule rule,
- Context ctx,
- Collector<Keyed<Transaction, String, Integer>> out) {
- BroadcastState<Integer, Rule> broadcastState = ctx.getBroadcastState(RULES_STATE_DESCRIPTOR);
- broadcastState.put(rule.getRuleId(), rule);
- }
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="nf">processBroadcastElement</span><span class="o">(</span><span class="n">Rule</span> <span class="n">rule</span><span class="o">,</span>
+ <span class="n">Context</span> <span class="n">ctx</span><span class="o">,</span>
+ <span class="n">Collector</span><span class="o">&lt;</span><span class="n">Keyed</span><span class="o">&lt;</span><span class="n">Transaction</span><span class="o">,</span> <span class="n">String</span><span class="o">,</span> <span class="n">Integer</ [...]
+ <span class="n">BroadcastState</span><span class="o">&lt;</span><span class="n">Integer</span><span class="o">,</span> <span class="n">Rule</span><span class="o">&gt;</span> <span class="n">broadcastState</span> <span class="o">=</span> <span class="n">ctx</span><span class=&quo [...]
+ <span class="n">broadcastState</span><span class="o">.</span><span class="na">put</span><span class="o">(</span><span class="n">rule</span><span class="o">.</span><span class="na">getRuleId</span><span class="o">(),</span> <span class="n">rule</span><span class="o">);</span>
+ <span class="o">}</span>
- @Override
- public void processElement(Transaction event,
- ReadOnlyContext ctx,
- Collector<Keyed<Transaction, String, Integer>> out){
- ReadOnlyBroadcastState<Integer, Rule> rulesState =
- ctx.getBroadcastState(RULES_STATE_DESCRIPTOR);
- for (Map.Entry<Integer, Rule> entry : rulesState.immutableEntries()) {
- final Rule rule = entry.getValue();
- out.collect(
- new Keyed<>(
- event, KeysExtractor.getKey(rule.getGroupingKeyNames(), event), rule.getRuleId()));
- }
- }
-}
-```
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="nf">processElement</span><span class="o">(</span><span class="n">Transaction</span> <span class="n">event</span><span class="o">,</span>
+ <span class="n">ReadOnlyContext</span> <span class="n">ctx</span><span class="o">,</span>
+ <span class="n">Collector</span><span class="o">&lt;</span><span class="n">Keyed</span><span class="o">&lt;</span><span class="n">Transaction</span><span class="o">,</span> <span class="n">String</span><span class="o">,</span> <span class="n">Integer</span>&l [...]
+ <span class="n">ReadOnlyBroadcastState</span><span class="o">&lt;</span><span class="n">Integer</span><span class="o">,</span> <span class="n">Rule</span><span class="o">&gt;</span> <span class="n">rulesState</span> <span class="o">=</span>
+ <span class="n">ctx</span><span class="o">.</span><span class="na">getBroadcastState</span><span class="o">(</span><span class="n">RULES_STATE_DESCRIPTOR</span><span class="o">);</span>
+ <span class="k">for</span> <span class="o">(</span><span class="n">Map</span><span class="o">.</span><span class="na">Entry</span><span class="o">&lt;</span><span class="n">Integer</span><span class="o">,</span> <span class="n">Rule</span><span class="o">&gt;</sp [...]
+ <span class="kd">final</span> <span class="n">Rule</span> <span class="n">rule</span> <span class="o">=</span> <span class="n">entry</span><span class="o">.</span><span class="na">getValue</span><span class="o">();</span>
+ <span class="n">out</span><span class="o">.</span><span class="na">collect</span><span class="o">(</span>
+ <span class="k">new</span> <span class="n">Keyed</span><span class="o">&lt;&gt;(</span>
+ <span class="n">event</span><span class="o">,</span> <span class="n">KeysExtractor</span><span class="o">.</span><span class="na">getKey</span><span class="o">(</span><span class="n">rule</span><span class="o">.</span><span class="na">getGroupingKeyNames</span><span class=" [...]
+ <span class="o">}</span>
+ <span class="o">}</span>
+<span class="o">}</span></code></pre></div>
-In the above code, `processElement()` receives Transactions, and `processBroadcastElement()` receives Rule updates. When a new rule is created, it is distributed as depicted in Figure 6 and saved in all parallel instances of the operator using `processBroadcastState`. We use a Rule's ID as the key to store and reference individual rules. Instead of iterating over a hardcoded `List<Rules>`, we iterate over entries in the dynamically-updated broadcast state.
+<p>In the above code, <code>processElement()</code> receives Transactions, and <code>processBroadcastElement()</code> receives Rule updates. When a new rule is created, it is distributed as depicted in Figure 6 and saved in all parallel instances of the operator using <code>processBroadcastState</code>. We use a Rule’s ID as the key to store and reference individual rules. Instead of iterating over a hardcoded <code>List&lt;Rules&gt [...]
-`DynamicAlertFunction` follows the same logic with respect to storing the rules in the broadcast `MapState`. As described in [Part 1](https://flink.apache.org/news/2020/01/15/demo-fraud-detection.html), each message in the `processElement` input is intended to be processed by one specific rule and comes "pre-marked" with a corresponding ID by `DynamicKeyFunction`. All we need to do is retrieve the definition of the corresponding rule from `BroadcastState` by using the provided [...]
+<p><code>DynamicAlertFunction</code> follows the same logic with respect to storing the rules in the broadcast <code>MapState</code>. As described in <a href="https://flink.apache.org/news/2020/01/15/demo-fraud-detection.html">Part 1</a>, each message in the <code>processElement</code> input is intended to be processed by one specific rule and comes “pre-marked” with a corresponding ID by <code>DynamicKeyFunction</ [...]
-# Summary
+<h1 id="summary">Summary</h1>
-In this blog post, we continued our investigation of the use case of a Fraud Detection System built with Apache Flink. We looked into different ways in which data can be distributed between parallel operator instances and, most importantly, examined broadcast state. We demonstrated how dynamic partitioning — a pattern described in the [first part](https://flink.apache.org/news/2020/01/15/demo-fraud-detection.html) of the series — can be combined and enhanced by the functionality provided [...]
+<p>In this blog post, we continued our investigation of the use case of a Fraud Detection System built with Apache Flink. We looked into different ways in which data can be distributed between parallel operator instances and, most importantly, examined broadcast state. We demonstrated how dynamic partitioning — a pattern described in the <a href="https://flink.apache.org/news/2020/01/15/demo-fraud-detection.html">first part</a> of the series — can be combined [...]
</description>
-<pubDate>Tue, 24 Mar 2020 12:00:00 +0000</pubDate>
+<pubDate>Tue, 24 Mar 2020 13:00:00 +0100</pubDate>
<link>https://flink.apache.org/news/2020/03/24/demo-fraud-detection-2.html</link>
<guid isPermaLink="true">/news/2020/03/24/demo-fraud-detection-2.html</guid>
</item>
<item>
<title>Apache Beam: How Beam Runs on Top of Flink</title>
-<description>Note: This blog post is based on the talk ["Beam on Flink: How Does It Actually Work?"](https://www.youtube.com/watch?v=hxHGLrshnCY).
+<description><p>Note: This blog post is based on the talk <a href="https://www.youtube.com/watch?v=hxHGLrshnCY">“Beam on Flink: How Does It Actually Work?”</a>.</p>
-[Apache Flink](https://flink.apache.org/) and [Apache Beam](https://beam.apache.org/) are open-source frameworks for parallel, distributed data processing at scale. Unlike Flink, Beam does not come with a full-blown execution engine of its own but plugs into other execution engines, such as Apache Flink, Apache Spark, or Google Cloud Dataflow. In this blog post we discuss the reasons to use Flink together with Beam for your batch and stream processing needs. We also take a closer look at [...]
+<p><a href="https://flink.apache.org/">Apache Flink</a> and <a href="https://beam.apache.org/">Apache Beam</a> are open-source frameworks for parallel, distributed data processing at scale. Unlike Flink, Beam does not come with a full-blown execution engine of its own but plugs into other execution engines, such as Apache Flink, Apache Spark, or Google Cloud Dataflow. In this blog post we discuss the reasons to use Flink together with Bea [...]
+<h1 id="what-is-apache-beam">What is Apache Beam</h1>
-# What is Apache Beam
+<p><a href="https://beam.apache.org/">Apache Beam</a> is an open-source, unified model for defining batch and streaming data-parallel processing pipelines. It is unified in the sense that you use a single API, in contrast to using a separate API for batch and streaming like it is the case in Flink. Beam was originally developed by Google which released it in 2014 as the Cloud Dataflow SDK. In 2016, it was donated to <a href="https://www.apache.org/&quo [...]
-[Apache Beam](https://beam.apache.org/) is an open-source, unified model for defining batch and streaming data-parallel processing pipelines. It is unified in the sense that you use a single API, in contrast to using a separate API for batch and streaming like it is the case in Flink. Beam was originally developed by Google which released it in 2014 as the Cloud Dataflow SDK. In 2016, it was donated to [the Apache Software Foundation](https://www.apache.org/) with the name of Beam. It ha [...]
+<p>The execution model, as well as the API of Apache Beam, are similar to Flink’s. Both frameworks are inspired by the <a href="https://static.googleusercontent.com/media/research.google.com/en//archive/mapreduce-osdi04.pdf">MapReduce</a>, <a href="https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/41378.pdf">MillWheel</a>, and <a href="https://research.google/pubs/pub43864/">Dataflow</a> [...]
-The execution model, as well as the API of Apache Beam, are similar to Flink's. Both frameworks are inspired by the [MapReduce](https://static.googleusercontent.com/media/research.google.com/en//archive/mapreduce-osdi04.pdf), [MillWheel](https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/41378.pdf), and [Dataflow](https://research.google/pubs/pub43864/) papers. Like Flink, Beam is designed for parallel, distributed data processing. Both have similar trans [...]
-
-One of the most exciting developments in the Beam technology is the framework’s support for multiple programming languages including Java, Python, Go, Scala and SQL. Essentially, developers can write their applications in a programming language of their choice. Beam, with the help of the Runners, translates the program to one of the execution engines, as shown in the diagram below.
+<p>One of the most exciting developments in the Beam technology is the framework’s support for multiple programming languages including Java, Python, Go, Scala and SQL. Essentially, developers can write their applications in a programming language of their choice. Beam, with the help of the Runners, translates the program to one of the execution engines, as shown in the diagram below.</p>
<center>
-<img src="{{ site.baseurl }}/img/blog/2020-02-22-beam-on-flink/flink-runner-beam-beam-vision.png" width="600px" alt="The vision of Apache Beam"/>
+<img src="/img/blog/2020-02-22-beam-on-flink/flink-runner-beam-beam-vision.png" width="600px" alt="The vision of Apache Beam" />
</center>
+<h1 id="reasons-to-use-beam-with-flink">Reasons to use Beam with Flink</h1>
-# Reasons to use Beam with Flink
-
-Why would you want to use Beam with Flink instead of directly using Flink? Ultimately, Beam and Flink complement each other and provide additional value to the user. The main reasons for using Beam with Flink are the following:
+<p>Why would you want to use Beam with Flink instead of directly using Flink? Ultimately, Beam and Flink complement each other and provide additional value to the user. The main reasons for using Beam with Flink are the following:</p>
-* Beam provides a unified API for both batch and streaming scenarios.
-* Beam comes with native support for different programming languages, like Python or Go with all their libraries like Numpy, Pandas, Tensorflow, or TFX.
-* You get the power of Apache Flink like its exactly-once semantics, strong memory management and robustness.
-* Beam programs run on your existing Flink infrastructure or infrastructure for other supported Runners, like Spark or Google Cloud Dataflow.
-* You get additional features like side inputs and cross-language pipelines that are not supported natively in Flink but only supported when using Beam with Flink.
-
-
-# The Flink Runner in Beam
+<ul>
+ <li>Beam provides a unified API for both batch and streaming scenarios.</li>
+ <li>Beam comes with native support for different programming languages, like Python or Go with all their libraries like Numpy, Pandas, Tensorflow, or TFX.</li>
+ <li>You get the power of Apache Flink like its exactly-once semantics, strong memory management and robustness.</li>
+ <li>Beam programs run on your existing Flink infrastructure or infrastructure for other supported Runners, like Spark or Google Cloud Dataflow.</li>
+ <li>You get additional features like side inputs and cross-language pipelines that are not supported natively in Flink but only supported when using Beam with Flink.</li>
+</ul>
-The Flink Runner in Beam translates Beam pipelines into Flink jobs. The translation can be parameterized using Beam's pipeline options which are parameters for settings like configuring the job name, parallelism, checkpointing, or metrics reporting.
+<h1 id="the-flink-runner-in-beam">The Flink Runner in Beam</h1>
-If you are familiar with a DataSet or a DataStream, you will have no problems understanding what a PCollection is. PCollection stands for parallel collection in Beam and is exactly what DataSet/DataStream would be in Flink. Due to Beam's unified API we only have one type of results of transformation: PCollection.
+<p>The Flink Runner in Beam translates Beam pipelines into Flink jobs. The translation can be parameterized using Beam’s pipeline options which are parameters for settings like configuring the job name, parallelism, checkpointing, or metrics reporting.</p>
-Beam pipelines are composed of transforms. Transforms are like operators in Flink and come in two flavors: primitive and composite transforms. The beauty of all this is that Beam only comes with a small set of primitive transforms which are:
+<p>If you are familiar with a DataSet or a DataStream, you will have no problems understanding what a PCollection is. PCollection stands for parallel collection in Beam and is exactly what DataSet/DataStream would be in Flink. Due to Beam’s unified API we only have one type of results of transformation: PCollection.</p>
-- `Source` (for loading data)
-- `ParDo` (think of a flat map operator on steroids)
-- `GroupByKey` (think of keyBy() in Flink)
-- `AssignWindows` (windows can be assigned at any point in time in Beam)
-- `Flatten` (like a union() operation in Flink)
+<p>Beam pipelines are composed of transforms. Transforms are like operators in Flink and come in two flavors: primitive and composite transforms. The beauty of all this is that Beam only comes with a small set of primitive transforms which are:</p>
-Composite transforms are built by combining the above primitive transforms. For example, `Combine = GroupByKey + ParDo`.
+<ul>
+ <li><code>Source</code> (for loading data)</li>
+ <li><code>ParDo</code> (think of a flat map operator on steroids)</li>
+ <li><code>GroupByKey</code> (think of keyBy() in Flink)</li>
+ <li><code>AssignWindows</code> (windows can be assigned at any point in time in Beam)</li>
+ <li><code>Flatten</code> (like a union() operation in Flink)</li>
+</ul>
+<p>Composite transforms are built by combining the above primitive transforms. For example, <code>Combine = GroupByKey + ParDo</code>.</p>
-# Flink Runner Internals
+<h1 id="flink-runner-internals">Flink Runner Internals</h1>
-Although using the Flink Runner in Beam has no prerequisite to understanding its internals, we provide more details of how the Flink runner works in Beam to share knowledge of how the two frameworks can integrate and work together to provide state-of-the-art streaming data pipelines.
+<p>Although using the Flink Runner in Beam has no prerequisite to understanding its internals, we provide more details of how the Flink runner works in Beam to share knowledge of how the two frameworks can integrate and work together to provide state-of-the-art streaming data pipelines.</p>
-The Flink Runner has two translation paths. Depending on whether we execute in batch or streaming mode, the Runner either translates into Flink's DataSet or into Flink's DataStream API. Since multi-language support has been added to Beam, another two translation paths have been added. To summarize the four modes:
+<p>The Flink Runner has two translation paths. Depending on whether we execute in batch or streaming mode, the Runner either translates into Flink’s DataSet or into Flink’s DataStream API. Since multi-language support has been added to Beam, another two translation paths have been added. To summarize the four modes:</p>
-1. **The Classic Flink Runner for batch jobs:** Executes batch Java pipelines
-2. **The Classic Flink Runner for streaming jobs:** Executes streaming Java pipelines
-3. **The Portable Flink Runner for batch jobs:** Executes Java as well as Python, Go and other supported SDK pipelines for batch scenarios
-4. **The Portable Flink Runner for streaming jobs:** Executes Java as well as Python, Go and other supported SDK pipelines for streaming scenarios
+<ol>
+ <li><strong>The Classic Flink Runner for batch jobs:</strong> Executes batch Java pipelines</li>
+ <li><strong>The Classic Flink Runner for streaming jobs:</strong> Executes streaming Java pipelines</li>
+ <li><strong>The Portable Flink Runner for batch jobs:</strong> Executes Java as well as Python, Go and other supported SDK pipelines for batch scenarios</li>
+ <li><strong>The Portable Flink Runner for streaming jobs:</strong> Executes Java as well as Python, Go and other supported SDK pipelines for streaming scenarios</li>
+</ol>
<center>
-<img src="{{ site.baseurl }}/img/blog/2020-02-22-beam-on-flink/flink-runner-beam-runner-translation-paths.png" width="300px" alt="The 4 translation paths in the Beam's Flink Runner"/>
+<img src="/img/blog/2020-02-22-beam-on-flink/flink-runner-beam-runner-translation-paths.png" width="300px" alt="The 4 translation paths in the Beam's Flink Runner" />
</center>
+<h2 id="the-classic-flink-runner-in-beam">The “Classic” Flink Runner in Beam</h2>
-## The “Classic” Flink Runner in Beam
-
-The classic Flink Runner was the initial version of the Runner, hence the "classic" name. Beam pipelines are represented as a graph in Java which is composed of the aforementioned composite and primitive transforms. Beam provides translators which traverse the graph in topological order. Topological order means that we start from all the sources first as we iterate through the graph. Presented with a transform from the graph, the Flink Runner generates the API calls as you woul [...]
+<p>The classic Flink Runner was the initial version of the Runner, hence the “classic” name. Beam pipelines are represented as a graph in Java which is composed of the aforementioned composite and primitive transforms. Beam provides translators which traverse the graph in topological order. Topological order means that we start from all the sources first as we iterate through the graph. Presented with a transform from the graph, the Flink Runner generates the API calls as you would [...]
<center>
-<img src="{{ site.baseurl }}/img/blog/2020-02-22-beam-on-flink/classic-flink-runner-beam.png" width="600px" alt="The Classic Flink Runner in Beam"/>
+<img src="/img/blog/2020-02-22-beam-on-flink/classic-flink-runner-beam.png" width="600px" alt="The Classic Flink Runner in Beam" />
</center>
-While Beam and Flink share very similar concepts, there are enough differences between the two frameworks that make Beam pipelines impossible to be translated 1:1 into a Flink program. In the following sections, we will present the key differences:
+<p>While Beam and Flink share very similar concepts, there are enough differences between the two frameworks that make Beam pipelines impossible to be translated 1:1 into a Flink program. In the following sections, we will present the key differences:</p>
-### Serializers vs Coders
+<h3 id="serializers-vs-coders">Serializers vs Coders</h3>
-When data is transferred over the wire in Flink, it has to be turned into bytes. This is done with the help of serializers. Flink has a type system to instantiate the correct coder for a given type, e.g. `StringTypeSerializer` for a String. Apache Beam also has its own type system which is similar to Flink's but uses slightly different interfaces. Serializers are called Coders in Beam. In order to make a Beam Coder run in Flink, we have to make the two serializer types compatible. Th [...]
+<p>When data is transferred over the wire in Flink, it has to be turned into bytes. This is done with the help of serializers. Flink has a type system to instantiate the correct coder for a given type, e.g. <code>StringTypeSerializer</code> for a String. Apache Beam also has its own type system which is similar to Flink’s but uses slightly different interfaces. Serializers are called Coders in Beam. In order to make a Beam Coder run in Flink, we have to make the two ser [...]
<center>
-<img src="{{ site.baseurl }}/img/blog/2020-02-22-beam-on-flink/flink-runner-beam-serializers-coders.png" width="300px" alt="Serializers vs Coders"/>
+<img src="/img/blog/2020-02-22-beam-on-flink/flink-runner-beam-serializers-coders.png" width="300px" alt="Serializers vs Coders" />
</center>
-### Read
-
-The `Read` transform provides a way to read data into your pipeline in Beam. The Read transform is supported by two wrappers in Beam, the `SourceInputFormat` for batch processing and the `UnboundedSourceWrapper` for stream processing.
+<h3 id="read">Read</h3>
-### ParDo
+<p>The <code>Read</code> transform provides a way to read data into your pipeline in Beam. The Read transform is supported by two wrappers in Beam, the <code>SourceInputFormat</code> for batch processing and the <code>UnboundedSourceWrapper</code> for stream processing.</p>
-`ParDo` is the swiss army knife of Beam and can be compared to a `RichFlatMapFunction` in Flink with additional features such as `SideInputs`, `SideOutputs`, State and Timers. `ParDo` is essentially translated by the Flink runner using the `FlinkDoFnFunction` for batch processing or the `FlinkStatefulDoFnFunction`, while for streaming scenarios the translation is executed with the `DoFnOperator` that takes care of checkpointing and buffering of data during checkpoints, watermark emission [...]
+<h3 id="pardo">ParDo</h3>
-### Side Inputs
+<p><code>ParDo</code> is the swiss army knife of Beam and can be compared to a <code>RichFlatMapFunction</code> in Flink with additional features such as <code>SideInputs</code>, <code>SideOutputs</code>, State and Timers. <code>ParDo</code> is essentially translated by the Flink runner using the <code>FlinkDoFnFunction</code> for batch processing or the <code>FlinkStatefulDoFnFunction</code>, while [...]
-In addition to the main input, ParDo transforms can have a number of side inputs. A side input can be a static set of data that you want to have available at all parallel instances. However, it is more flexible than that. You can have keyed and even windowed side input which updates based on the window size. This is a very powerful concept which does not exist in Flink but is added on top of Flink using Beam.
+<h3 id="side-inputs">Side Inputs</h3>
-### AssignWindows
+<p>In addition to the main input, ParDo transforms can have a number of side inputs. A side input can be a static set of data that you want to have available at all parallel instances. However, it is more flexible than that. You can have keyed and even windowed side input which updates based on the window size. This is a very powerful concept which does not exist in Flink but is added on top of Flink using Beam.</p>
-In Flink, windows are assigned by the `WindowOperator` when you use the `window()` in the API. In Beam, windows can be assigned at any point in time. Any element is implicitly part of a window. If no window is assigned explicitly, the element is part of the `GlobalWindow`. Window information is stored for each element in a wrapper called `WindowedValue`. The window information is only used once we issue a `GroupByKey`.
+<h3 id="assignwindows">AssignWindows</h3>
-### GroupByKey
+<p>In Flink, windows are assigned by the <code>WindowOperator</code> when you use the <code>window()</code> in the API. In Beam, windows can be assigned at any point in time. Any element is implicitly part of a window. If no window is assigned explicitly, the element is part of the <code>GlobalWindow</code>. Window information is stored for each element in a wrapper called <code>WindowedValue</code>. The window information is only use [...]
-Most of the time it is useful to partition the data by a key. In Flink, this is done via the `keyBy()` API call. In Beam the `GroupByKey` transform can only be applied if the input is of the form `KV<Key, Value>`. Unlike Flink where the key can even be nested inside the data, Beam enforces the key to always be explicit. The `GroupByKey` transform then groups the data by key and by window which is similar to what `keyBy(..).window(..)` would give us in Flink. Beam has its own set of [...]
+<h3 id="groupbykey">GroupByKey</h3>
-### Flatten
+<p>Most of the time it is useful to partition the data by a key. In Flink, this is done via the <code>keyBy()</code> API call. In Beam the <code>GroupByKey</code> transform can only be applied if the input is of the form <code>KV&lt;Key, Value&gt;</code>. Unlike Flink where the key can even be nested inside the data, Beam enforces the key to always be explicit. The <code>GroupByKey</code> transform then groups the data by key [...]
-The Flatten operator takes multiple DataSet/DataStreams, called P[arallel]Collections in Beam, and combines them into one collection. This is equivalent to Flink's `union()` operation.
+<h3 id="flatten">Flatten</h3>
+<p>The Flatten operator takes multiple DataSet/DataStreams, called P[arallel]Collections in Beam, and combines them into one collection. This is equivalent to Flink’s <code>union()</code> operation.</p>
-## The “Portable” Flink Runner in Beam
+<h2 id="the-portable-flink-runner-in-beam">The “Portable” Flink Runner in Beam</h2>
-The portable Flink Runner in Beam is the evolution of the classic Runner. Classic Runners are tied to the JVM ecosystem, but the Beam community wanted to move past this and also execute Python, Go and other languages. This adds another dimension to Beam in terms of portability because, like previously mentioned, Beam already had portability across execution engines. It was necessary to change the translation logic of the Runner to be able to support language portability.
+<p>The portable Flink Runner in Beam is the evolution of the classic Runner. Classic Runners are tied to the JVM ecosystem, but the Beam community wanted to move past this and also execute Python, Go and other languages. This adds another dimension to Beam in terms of portability because, like previously mentioned, Beam already had portability across execution engines. It was necessary to change the translation logic of the Runner to be able to support language portability.</p>
-There are two important building blocks for portable Runners:
+<p>There are two important building blocks for portable Runners:</p>
-1. A common pipeline format across all the languages: The Runner API
-2. A common interface during execution for the communication between the Runner and the code written in any language: The Fn API
+<ol>
+ <li>A common pipeline format across all the languages: The Runner API</li>
+ <li>A common interface during execution for the communication between the Runner and the code written in any language: The Fn API</li>
+</ol>
-The Runner API provides a universal representation of the pipeline as Protobuf which contains the transforms, types, and user code. Protobuf was chosen as the format because every language has libraries available for it. Similarly, for the execution part, Beam introduced the Fn API interface to handle the communication between the Runner/execution engine and the user code that may be written in a different language and executes in a different process. Fn API is pronounced "fun API&q [...]
+<p>The Runner API provides a universal representation of the pipeline as Protobuf which contains the transforms, types, and user code. Protobuf was chosen as the format because every language has libraries available for it. Similarly, for the execution part, Beam introduced the Fn API interface to handle the communication between the Runner/execution engine and the user code that may be written in a different language and executes in a different process. Fn API is pronounced “fun A [...]
<center>
-<img src="{{ site.baseurl }}/img/blog/2020-02-22-beam-on-flink/flink-runner-beam-language-portability.png" width="600px" alt="Language Portability in Apache Beam"/>
+<img src="/img/blog/2020-02-22-beam-on-flink/flink-runner-beam-language-portability.png" width="600px" alt="Language Portability in Apache Beam" />
</center>
+<h2 id="how-are-beam-programs-translated-in-language-portability">How Are Beam Programs Translated In Language Portability?</h2>
-## How Are Beam Programs Translated In Language Portability?
+<p>Users write their Beam pipelines in one language, but they may get executed in an environment based on a completely different language. How does that work? To explain that, let’s follow the lifecycle of a pipeline. Let’s suppose we use the Python SDK to write the pipeline. Before submitting the pipeline via the Job API to Beam’s JobServer, Beam would convert it to the Runner API, the language-agnostic format we described before. The JobServer is also a Beam component that handle [...]
-Users write their Beam pipelines in one language, but they may get executed in an environment based on a completely different language. How does that work? To explain that, let's follow the lifecycle of a pipeline. Let's suppose we use the Python SDK to write the pipeline. Before submitting the pipeline via the Job API to Beam's JobServer, Beam would convert it to the Runner API, the language-agnostic format we described before. The JobServer is also a Beam component that han [...]
-
-- Docker-based (the default)
-- Process-based (a simple process is started)
-- Externally-provided (K8s or other schedulers)
-- Embedded (intended for testing and only works with Java)
+<ul>
+ <li>Docker-based (the default)</li>
+ <li>Process-based (a simple process is started)</li>
+ <li>Externally-provided (K8s or other schedulers)</li>
+ <li>Embedded (intended for testing and only works with Java)</li>
+</ul>
-Environments hold the _SDK Harness_ which is the code that handles the execution and the communication with the Runner over the Fn API. For example, when Flink executes Python code, it sends the data to the Python environment containing the Python SDK Harness. Sending data to an external process involves a minor overhead which we have measured to be 5-10% slower than the classic Java pipelines. However, Beam uses a fusion of transforms to execute as many transforms as possible in the sam [...]
+<p>Environments hold the <em>SDK Harness</em> which is the code that handles the execution and the communication with the Runner over the Fn API. For example, when Flink executes Python code, it sends the data to the Python environment containing the Python SDK Harness. Sending data to an external process involves a minor overhead which we have measured to be 5-10% slower than the classic Java pipelines. However, Beam uses a fusion of transforms to execute as many trans [...]
<center>
-<img src="{{ site.baseurl }}/img/blog/2020-02-22-beam-on-flink/flink-runner-beam-language-portability-architecture.png" width="600px" alt="Language Portability Architecture in beam"/>
+<img src="/img/blog/2020-02-22-beam-on-flink/flink-runner-beam-language-portability-architecture.png" width="600px" alt="Language Portability Architecture in beam" />
</center>
+<p>Environments can be present for many languages. This opens up an entirely new type of pipelines: cross-language pipelines. In cross-language pipelines we can combine transforms of two or more languages, e.g. a machine learning pipeline with the feature generation written in Java and the learning written in Python. All this can be run on top of Flink.</p>
-Environments can be present for many languages. This opens up an entirely new type of pipelines: cross-language pipelines. In cross-language pipelines we can combine transforms of two or more languages, e.g. a machine learning pipeline with the feature generation written in Java and the learning written in Python. All this can be run on top of Flink.
-
+<h2 id="conclusion">Conclusion</h2>
-## Conclusion
+<p>Using Apache Beam with Apache Flink combines (a.) the power of Flink with (b.) the flexibility of Beam. All it takes to run Beam is a Flink cluster, which you may already have. Apache Beam’s fully-fledged Python API is probably the most compelling argument for using Beam with Flink, but the unified API which allows to “write-once” and “execute-anywhere” is also very appealing to Beam users. On top of this, features like side inputs and a rich connector ecosystem are also reason [...]
-Using Apache Beam with Apache Flink combines (a.) the power of Flink with (b.) the flexibility of Beam. All it takes to run Beam is a Flink cluster, which you may already have. Apache Beam's fully-fledged Python API is probably the most compelling argument for using Beam with Flink, but the unified API which allows to "write-once" and "execute-anywhere" is also very appealing to Beam users. On top of this, features like side inputs and a rich connector ecosystem [...]
+<p>With the introduction of schemas, a new format for handling type information, Beam is heading in a similar direction as Flink with its type system which is essential for the Table API or SQL. Speaking of, the next Flink release will include a Python version of the Table API which is based on the language portability of Beam. Looking ahead, the Beam community plans to extend the support for interactive programs like notebooks. TFX, which is built with Beam, is a very powerful way [...]
-With the introduction of schemas, a new format for handling type information, Beam is heading in a similar direction as Flink with its type system which is essential for the Table API or SQL. Speaking of, the next Flink release will include a Python version of the Table API which is based on the language portability of Beam. Looking ahead, the Beam community plans to extend the support for interactive programs like notebooks. TFX, which is built with Beam, is a very powerful way to solve [...]
-
-For many years, Beam and Flink have inspired and learned from each other. With the Python support being based on Beam in Flink, they only seem to come closer to each other. That's all the better for the community, and also users have more options and functionality to choose from.
+<p>For many years, Beam and Flink have inspired and learned from each other. With the Python support being based on Beam in Flink, they only seem to come closer to each other. That’s all the better for the community, and also users have more options and functionality to choose from.</p>
</description>
-<pubDate>Sat, 22 Feb 2020 12:00:00 +0000</pubDate>
+<pubDate>Sat, 22 Feb 2020 13:00:00 +0100</pubDate>
<link>https://flink.apache.org/ecosystem/2020/02/22/apache-beam-how-beam-runs-on-top-of-flink.html</link>
<guid isPermaLink="true">/ecosystem/2020/02/22/apache-beam-how-beam-runs-on-top-of-flink.html</guid>
</item>
<item>
<title>No Java Required: Configuring Sources and Sinks in SQL</title>
-<description># Introduction
+<description><h1 id="introduction">Introduction</h1>
-The recent [Apache Flink 1.10 release](https://flink.apache.org/news/2020/02/11/release-1.10.0.html) includes many exciting features.
-In particular, it marks the end of the community's year-long effort to merge in the [Blink SQL contribution](https://flink.apache.org/news/2019/02/13/unified-batch-streaming-blink.html) from Alibaba.
+<p>The recent <a href="https://flink.apache.org/news/2020/02/11/release-1.10.0.html">Apache Flink 1.10 release</a> includes many exciting features.
+In particular, it marks the end of the community’s year-long effort to merge in the <a href="https://flink.apache.org/news/2019/02/13/unified-batch-streaming-blink.html">Blink SQL contribution</a> from Alibaba.
The reason the community chose to spend so much time on the contribution is that SQL works.
It allows Flink to offer a truly unified interface over batch and streaming and makes stream processing accessible to a broad audience of developers and analysts.
-Best of all, Flink SQL is ANSI-SQL compliant, which means if you've ever used a database in the past, you already know it[^1]!
+Best of all, Flink SQL is ANSI-SQL compliant, which means if you’ve ever used a database in the past, you already know it<sup id="fnref:1"><a href="#fn:1" class="footnote">1</a></sup>!</p>
-A lot of work focused on improving runtime performance and progressively extending its coverage of the SQL standard.
+<p>A lot of work focused on improving runtime performance and progressively extending its coverage of the SQL standard.
Flink now supports the full TPC-DS query set for batch queries, reflecting the readiness of its SQL engine to address the needs of modern data warehouse-like workloads.
-Its streaming SQL supports an almost equal set of features - those that are well defined on a streaming runtime - including [complex joins](https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/streaming/joins.html) and [MATCH_RECOGNIZE](https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/streaming/match_recognize.html).
+Its streaming SQL supports an almost equal set of features - those that are well defined on a streaming runtime - including <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/streaming/joins.html">complex joins</a> and <a href="https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/streaming/match_recognize.html">MATCH_RECOGNIZE</a>.</p>
-As important as this work is, the community also strives to make these features generally accessible to the broadest audience possible.
-That is why the Flink community is excited in 1.10 to offer production-ready DDL syntax (e.g., `CREATE TABLE`, `DROP TABLE`) and a refactored catalog interface.
+<p>As important as this work is, the community also strives to make these features generally accessible to the broadest audience possible.
+That is why the Flink community is excited in 1.10 to offer production-ready DDL syntax (e.g., <code>CREATE TABLE</code>, <code>DROP TABLE</code>) and a refactored catalog interface.</p>
-# Accessing Your Data Where It Lives
+<h1 id="accessing-your-data-where-it-lives">Accessing Your Data Where It Lives</h1>
-Flink does not store data at rest; it is a compute engine and requires other systems to consume input from and write its output.
-Those that have used Flink's `DataStream` API in the past will be familiar with connectors that allow for interacting with external systems.
-Flink has a vast connector ecosystem that includes all major message queues, filesystems, and databases.
+<p>Flink does not store data at rest; it is a compute engine and requires other systems to consume input from and write its output.
+Those that have used Flink’s <code>DataStream</code> API in the past will be familiar with connectors that allow for interacting with external systems.
+Flink has a vast connector ecosystem that includes all major message queues, filesystems, and databases.</p>
<div class="alert alert-info">
If your favorite system does not have a connector maintained in the central Apache Flink repository, check out the <a href="https://flink-packages.org">flink packages website</a>, which has a growing number of community-maintained components.
</div>
-While these connectors are battle-tested and production-ready, they are written in Java and configured in code, which means they are not amenable to pure SQL or Table applications.
-For a holistic SQL experience, not only queries need to be written in SQL, but also table definitions.
+<p>While these connectors are battle-tested and production-ready, they are written in Java and configured in code, which means they are not amenable to pure SQL or Table applications.
+For a holistic SQL experience, not only queries need to be written in SQL, but also table definitions.</p>
-# CREATE TABLE Statements
+<h1 id="create-table-statements">CREATE TABLE Statements</h1>
-While Flink SQL has long provided table abstractions atop some of Flink's most popular connectors, configurations were not always so straightforward.
-Beginning in 1.10, Flink supports defining tables through `CREATE TABLE` statements.
-With this feature, users can now create logical tables, backed by various external systems, in pure SQL.
+<p>While Flink SQL has long provided table abstractions atop some of Flink’s most popular connectors, configurations were not always so straightforward.
+Beginning in 1.10, Flink supports defining tables through <code>CREATE TABLE</code> statements.
+With this feature, users can now create logical tables, backed by various external systems, in pure SQL.</p>
-By defining tables in SQL, developers can write queries against logical schemas that are abstracted away from the underlying physical data store. Coupled with Flink SQL's unified approach to batch and stream processing, Flink provides a straight line from discovery to production.
+<p>By defining tables in SQL, developers can write queries against logical schemas that are abstracted away from the underlying physical data store. Coupled with Flink SQL’s unified approach to batch and stream processing, Flink provides a straight line from discovery to production.</p>
-Users can define tables over static data sets, anything from a local CSV file to a full-fledged data lake or even Hive.
-Leveraging Flink's efficient batch processing capabilities, they can perform ad-hoc queries searching for exciting insights.
+<p>Users can define tables over static data sets, anything from a local CSV file to a full-fledged data lake or even Hive.
+Leveraging Flink’s efficient batch processing capabilities, they can perform ad-hoc queries searching for exciting insights.
Once something interesting is identified, businesses can gain real-time and continuous insights by merely altering the table so that it is powered by a message queue such as Kafka.
-Because Flink guarantees SQL queries have unified semantics over batch and streaming, users can be confident that redeploying this query as a continuous streaming application over a message queue will output identical results.
-
-{% highlight sql %}
--- Define a table called orders that is backed by a Kafka topic
--- The definition includes all relevant Kafka properties,
--- the underlying format (JSON) and even defines a
--- watermarking algorithm based on one of the fields
--- so that this table can be used with event time.
-CREATE TABLE orders (
- user_id BIGINT,
- product STRING,
- order_time TIMESTAMP(3),
- WATERMARK FOR order_time AS order_time - '5' SECONDS
-) WITH (
- 'connector.type' = 'kafka',
- 'connector.version' = 'universal',
- 'connector.topic' = 'orders',
- 'connector.startup-mode' = 'earliest-offset',
- 'connector.properties.bootstrap.servers' = 'localhost:9092',
- 'format.type' = 'json'
-);
-
--- Define a table called product_analysis
--- on top of ElasticSearch 7 where we
--- can write the results of our query.
-CREATE TABLE product_analysis (
- product STRING,
- tracking_time TIMESTAMP(3),
- units_sold BIGINT
-) WITH (
- 'connector.type' = 'elasticsearch',
- 'connector.version' = '7',
- 'connector.hosts' = 'localhost:9200',
- 'connector.index' = 'ProductAnalysis',
- 'connector.document.type' = 'analysis'
-);
-
--- A simple query that analyzes order data
--- from Kafka and writes results into
--- ElasticSearch.
-INSERT INTO product_analysis
-SELECT
- product_id,
- TUMBLE_START(order_time, INTERVAL '1' DAY) as tracking_time,
- COUNT(*) as units_sold
-FROM orders
-GROUP BY
- product_id,
- TUMBLE(order_time, INTERVAL '1' DAY);
-{% endhighlight %}
-
-# Catalogs
-
-While being able to create tables is important, it often isn't enough.
-A business analyst, for example, shouldn't have to know what properties to set for Kafka, or even have to know what the underlying data source is, to be able to write a query.
-
-To solve this problem, Flink 1.10 also ships with a revamped catalog system for managing metadata about tables and user definined functions.
+Because Flink guarantees SQL queries have unified semantics over batch and streaming, users can be confident that redeploying this query as a continuous streaming application over a message queue will output identical results.</p>
+
+<figure class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="c1">-- Define a table called orders that is backed by a Kafka topic</span>
+<span class="c1">-- The definition includes all relevant Kafka properties,</span>
+<span class="c1">-- the underlying format (JSON) and even defines a</span>
+<span class="c1">-- watermarking algorithm based on one of the fields</span>
+<span class="c1">-- so that this table can be used with event time.</span>
+<span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">orders</span> <span class="p">(</span>
+ <span class="n">user_id</span> <span class="nb">BIGINT</span><span class="p">,</span>
+ <span class="n">product</span> <span class="n">STRING</span><span class="p">,</span>
+ <span class="n">order_time</span> <span class="k">TIMESTAMP</span><span class="p">(</span><span class="mi">3</span><span class="p">),</span>
+ <span class="n">WATERMARK</span> <span class="k">FOR</span> <span class="n">order_time</span> <span class="k">AS</span> <span class="n">order_time</span> <span class="o">-</span> <span class="s1">&#39;5&#39;</span> <span class="n">SECONDS</span>
+<span class="p">)</span> <span class="k">WITH</span> <span class="p">(</span>
+ <span class="s1">&#39;connector.type&#39;</span> <span class="o">=</span> <span class="s1">&#39;kafka&#39;</span><span class="p">,</span>
+ <span class="s1">&#39;connector.version&#39;</span> <span class="o">=</span> <span class="s1">&#39;universal&#39;</span><span class="p">,</span>
+ <span class="s1">&#39;connector.topic&#39;</span> <span class="o">=</span> <span class="s1">&#39;orders&#39;</span><span class="p">,</span>
+ <span class="s1">&#39;connector.startup-mode&#39;</span> <span class="o">=</span> <span class="s1">&#39;earliest-offset&#39;</span><span class="p">,</span>
+ <span class="s1">&#39;connector.properties.bootstrap.servers&#39;</span> <span class="o">=</span> <span class="s1">&#39;localhost:9092&#39;</span><span class="p">,</span>
+ <span class="s1">&#39;format.type&#39;</span> <span class="o">=</span> <span class="s1">&#39;json&#39;</span>
+<span class="p">);</span>
+
+<span class="c1">-- Define a table called product_analysis</span>
+<span class="c1">-- on top of ElasticSearch 7 where we </span>
+<span class="c1">-- can write the results of our query. </span>
+<span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">product_analysis</span> <span class="p">(</span>
+ <span class="n">product</span> <span class="n">STRING</span><span class="p">,</span>
+ <span class="n">tracking_time</span> <span class="k">TIMESTAMP</span><span class="p">(</span><span class="mi">3</span><span class="p">),</span>
+ <span class="n">units_sold</span> <span class="nb">BIGINT</span>
+<span class="p">)</span> <span class="k">WITH</span> <span class="p">(</span>
+ <span class="s1">&#39;connector.type&#39;</span> <span class="o">=</span> <span class="s1">&#39;elasticsearch&#39;</span><span class="p">,</span>
+ <span class="s1">&#39;connector.version&#39;</span> <span class="o">=</span> <span class="s1">&#39;7&#39;</span><span class="p">,</span>
+ <span class="s1">&#39;connector.hosts&#39;</span> <span class="o">=</span> <span class="s1">&#39;localhost:9200&#39;</span><span class="p">,</span>
+ <span class="s1">&#39;connector.index&#39;</span> <span class="o">=</span> <span class="s1">&#39;ProductAnalysis&#39;</span><span class="p">,</span>
+ <span class="s1">&#39;connector.document.type&#39;</span> <span class="o">=</span> <span class="s1">&#39;analysis&#39;</span>
+<span class="p">);</span>
+
+<span class="c1">-- A simple query that analyzes order data</span>
+<span class="c1">-- from Kafka and writes results into </span>
+<span class="c1">-- ElasticSearch. </span>
+<span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">product_analysis</span>
+<span class="k">SELECT</span>
+ <span class="n">product_id</span><span class="p">,</span>
+ <span class="n">TUMBLE_START</span><span class="p">(</span><span class="n">order_time</span><span class="p">,</span> <span class="nb">INTERVAL</span> <span class="s1">&#39;1&#39;</span> <span class="k">DAY</span><span class="p">)</span> <span class="k">as</span> <span class="n [...]
+ <span class="k">COUNT</span><span class="p">(</span><span class="o">*</span><span class="p">)</span> <span class="k">as</span> <span class="n">units_sold</span>
+<span class="k">FROM</span> <span class="n">orders</span>
+<span class="k">GROUP</span> <span class="k">BY</span>
+ <span class="n">product_id</span><span class="p">,</span>
+ <span class="n">TUMBLE</span><span class="p">(</span><span class="n">order_time</span><span class="p">,</span> <span class="nb">INTERVAL</span> <span class="s1">&#39;1&#39;</span> <span class="k">DAY</span><span class="p">);</span></code></pre></figure>
+
+<h1 id="catalogs">Catalogs</h1>
+
+<p>While being able to create tables is important, it often isn’t enough.
+A business analyst, for example, shouldn’t have to know what properties to set for Kafka, or even have to know what the underlying data source is, to be able to write a query.</p>
+
+<p>To solve this problem, Flink 1.10 also ships with a revamped catalog system for managing metadata about tables and user definined functions.
With catalogs, users can create tables once and reuse them across Jobs and Sessions.
-Now, the team managing a data set can create a table and immediately make it accessible to other groups within their organization.
+Now, the team managing a data set can create a table and immediately make it accessible to other groups within their organization.</p>
-The most notable catalog that Flink integrates with today is Hive Metastore.
+<p>The most notable catalog that Flink integrates with today is Hive Metastore.
The Hive catalog allows Flink to fully interoperate with Hive and serve as a more efficient query engine.
-Flink supports reading and writing Hive tables, using Hive UDFs, and even leveraging Hive's metastore catalog to persist Flink specific metadata.
+Flink supports reading and writing Hive tables, using Hive UDFs, and even leveraging Hive’s metastore catalog to persist Flink specific metadata.</p>
-# Looking Ahead
+<h1 id="looking-ahead">Looking Ahead</h1>
-Flink SQL has made enormous strides to democratize stream processing, and 1.10 marks a significant milestone in that development.
+<p>Flink SQL has made enormous strides to democratize stream processing, and 1.10 marks a significant milestone in that development.
However, we are not ones to rest on our laurels and, the community is committed to raising the bar on standards while lowering the barriers to entry.
The community is looking to add more catalogs, such as JDBC and Apache Pulsar.
-We encourage you to sign up for the [mailing list](https://flink.apache.org/community.html) and stay on top of the announcements and new features in upcoming releases.
+We encourage you to sign up for the <a href="https://flink.apache.org/community.html">mailing list</a> and stay on top of the announcements and new features in upcoming releases.</p>
----
+<hr />
-[^1]: My colleague Timo, whose worked on Flink SQL from the beginning, has the entire SQL standard printed on his desk and references it before any changes are merged. It's enormous.
+<div class="footnotes">
+ <ol>
+ <li id="fn:1">
+ <p>My colleague Timo, whose worked on Flink SQL from the beginning, has the entire SQL standard printed on his desk and references it before any changes are merged. It’s enormous. <a href="#fnref:1" class="reversefootnote">&#8617;</a></p>
+ </li>
+ </ol>
+</div>
</description>
-<pubDate>Thu, 20 Feb 2020 12:00:00 +0000</pubDate>
+<pubDate>Thu, 20 Feb 2020 13:00:00 +0100</pubDate>
<link>https://flink.apache.org/news/2020/02/20/ddl.html</link>
<guid isPermaLink="true">/news/2020/02/20/ddl.html</guid>
</item>
<item>
<title>Apache Flink 1.10.0 Release Announcement</title>
-<description>The Apache Flink community is excited to hit the double digits and announce the release of Flink 1.10.0! As a result of the biggest community effort to date, with over 1.2k issues implemented and more than 200 contributors, this release introduces significant improvements to the overall performance and stability of Flink jobs, a preview of native Kubernetes integration and great advances in Python support (PyFlink).
-
-Flink 1.10 also marks the completion of the [Blink integration](https://flink.apache.org/news/2019/08/22/release-1.9.0.html#preview-of-the-new-blink-sql-query-processor), hardening streaming SQL and bringing mature batch processing to Flink with production-ready Hive integration and TPC-DS coverage. This blog post describes all major new features and improvements, important changes to be aware of and what to expect moving forward.
-
-{% toc %}
-
-The binary distribution and source artifacts are now available on the updated [Downloads page]({{ site.baseurl }}/downloads.html) of the Flink website. For more details, check the complete [release changelog](https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12345845) and the [updated documentation]({{ site.DOCS_BASE_URL }}flink-docs-release-1.10/). We encourage you to download the release and share your feedback with the community through the [Flink m [...]
-
-
-## New Features and Improvements
+<description><p>The Apache Flink community is excited to hit the double digits and announce the release of Flink 1.10.0! As a result of the biggest community effort to date, with over 1.2k issues implemented and more than 200 contributors, this release introduces significant improvements to the overall performance and stability of Flink jobs, a preview of native Kubernetes integration and great advances in Python support (PyFlink).</p>
+
+<p>Flink 1.10 also marks the completion of the <a href="https://flink.apache.org/news/2019/08/22/release-1.9.0.html#preview-of-the-new-blink-sql-query-processor">Blink integration</a>, hardening streaming SQL and bringing mature batch processing to Flink with production-ready Hive integration and TPC-DS coverage. This blog post describes all major new features and improvements, important changes to be aware of and what to expect moving forward.</p>
+
+<div class="page-toc">
+<ul id="markdown-toc">
+ <li><a href="#new-features-and-improvements" id="markdown-toc-new-features-and-improvements">New Features and Improvements</a> <ul>
+ <li><a href="#improved-memory-management-and-configuration" id="markdown-toc-improved-memory-management-and-configuration">Improved Memory Management and Configuration</a></li>
+ <li><a href="#unified-logic-for-job-submission" id="markdown-toc-unified-logic-for-job-submission">Unified Logic for Job Submission</a></li>
+ <li><a href="#native-kubernetes-integration-beta" id="markdown-toc-native-kubernetes-integration-beta">Native Kubernetes Integration (Beta)</a></li>
+ <li><a href="#table-apisql-production-ready-hive-integration" id="markdown-toc-table-apisql-production-ready-hive-integration">Table API/SQL: Production-ready Hive Integration</a></li>
+ <li><a href="#other-improvements-to-the-table-apisql" id="markdown-toc-other-improvements-to-the-table-apisql">Other Improvements to the Table API/SQL</a></li>
+ <li><a href="#pyflink-support-for-native-user-defined-functions-udfs" id="markdown-toc-pyflink-support-for-native-user-defined-functions-udfs">PyFlink: Support for Native User Defined Functions (UDFs)</a></li>
+ </ul>
+ </li>
+ <li><a href="#important-changes" id="markdown-toc-important-changes">Important Changes</a></li>
+ <li><a href="#release-notes" id="markdown-toc-release-notes">Release Notes</a></li>
+ <li><a href="#list-of-contributors" id="markdown-toc-list-of-contributors">List of Contributors</a></li>
+</ul>
+</div>
-### Improved Memory Management and Configuration
+<p>The binary distribution and source artifacts are now available on the updated <a href="/downloads.html">Downloads page</a> of the Flink website. For more details, check the complete <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&amp;version=12345845">release changelog</a> and the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/">updated documentation</a> [...]
-The current `TaskExecutor` memory configuration in Flink has some shortcomings that make it hard to reason about or optimize resource utilization, such as:
+<h2 id="new-features-and-improvements">New Features and Improvements</h2>
-* Different configuration models for memory footprint in Streaming and Batch execution;
+<h3 id="improved-memory-management-and-configuration">Improved Memory Management and Configuration</h3>
-* Complex and user-dependent configuration of off-heap state backends (i.e. RocksDB) in Streaming execution.
+<p>The current <code>TaskExecutor</code> memory configuration in Flink has some shortcomings that make it hard to reason about or optimize resource utilization, such as:</p>
-To make memory options more explicit and intuitive to users, Flink 1.10 introduces significant changes to the `TaskExecutor` memory model and configuration logic ([FLIP-49](https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors)). These changes make Flink more adaptable to all kinds of deployment environments (e.g. Kubernetes, Yarn, Mesos), giving users strict control over its memory consumption.
+<ul>
+ <li>
+ <p>Different configuration models for memory footprint in Streaming and Batch execution;</p>
+ </li>
+ <li>
+ <p>Complex and user-dependent configuration of off-heap state backends (i.e. RocksDB) in Streaming execution.</p>
+ </li>
+</ul>
-**Managed Memory Extension**
+<p>To make memory options more explicit and intuitive to users, Flink 1.10 introduces significant changes to the <code>TaskExecutor</code> memory model and configuration logic (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors">FLIP-49</a>). These changes make Flink more adaptable to all kinds of deployment environments (e.g. Kubernetes, Yarn, Mesos), giving users strict control ove [...]
-Managed memory was extended to also account for memory usage of `RocksDBStateBackend`. While batch jobs can use either on-heap or off-heap memory, streaming jobs with `RocksDBStateBackend` can use off-heap memory only. Therefore, to allow users to switch between Streaming and Batch execution without having to modify cluster configurations, managed memory is now always off-heap.
+<p><strong>Managed Memory Extension</strong></p>
-**Simplified RocksDB Configuration**
+<p>Managed memory was extended to also account for memory usage of <code>RocksDBStateBackend</code>. While batch jobs can use either on-heap or off-heap memory, streaming jobs with <code>RocksDBStateBackend</code> can use off-heap memory only. Therefore, to allow users to switch between Streaming and Batch execution without having to modify cluster configurations, managed memory is now always off-heap.</p>
-Configuring an off-heap state backend like RocksDB used to involve a good deal of manual tuning, like decreasing the JVM heap size or setting Flink to use off-heap memory. This can now be achieved through Flink's out-of-box configuration, and adjusting the memory budget for `RocksDBStateBackend` is as simple as resizing the managed memory size.
+<p><strong>Simplified RocksDB Configuration</strong></p>
-Another important improvement was to allow Flink to bind RocksDB native memory usage ([FLINK-7289](https://issues.apache.org/jira/browse/FLINK-7289)), preventing it from exceeding its total memory budget — this is especially relevant in containerized environments like Kubernetes. For details on how to enable and tune this feature, refer to [Tuning RocksDB]({{ site.DOCS_BASE_URL }}flink-docs-release-1.10/ops/state/large_state_tuning.html#tuning-rocksdb).
+<p>Configuring an off-heap state backend like RocksDB used to involve a good deal of manual tuning, like decreasing the JVM heap size or setting Flink to use off-heap memory. This can now be achieved through Flink’s out-of-box configuration, and adjusting the memory budget for <code>RocksDBStateBackend</code> is as simple as resizing the managed memory size.</p>
-<span class="label label-danger">Note</span> FLIP-49 changes the process of cluster resource configuration, which may require tuning your clusters for upgrades from previous Flink versions. For a comprehensive overview of the changes introduced and tuning guidance, consult [this setup]({{ site.DOCS_BASE_URL }}flink-docs-release-1.10/ops/memory/mem_setup.html).
+<p>Another important improvement was to allow Flink to bind RocksDB native memory usage (<a href="https://issues.apache.org/jira/browse/FLINK-7289">FLINK-7289</a>), preventing it from exceeding its total memory budget — this is especially relevant in containerized environments like Kubernetes. For details on how to enable and tune this feature, refer to <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/state/large_state_tuning.h [...]
+<p><span class="label label-danger">Note</span> FLIP-49 changes the process of cluster resource configuration, which may require tuning your clusters for upgrades from previous Flink versions. For a comprehensive overview of the changes introduced and tuning guidance, consult <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/memory/mem_setup.html">this setup</a>.</p>
-### Unified Logic for Job Submission
+<h3 id="unified-logic-for-job-submission">Unified Logic for Job Submission</h3>
-Prior to this release, job submission was part of the duties of the Execution Environments and closely tied to the different deployment targets (e.g. Yarn, Kubernetes, Mesos). This led to a poor separation of concerns and, over time, to a growing number of customized environments that users needed to configure and manage separately.
+<p>Prior to this release, job submission was part of the duties of the Execution Environments and closely tied to the different deployment targets (e.g. Yarn, Kubernetes, Mesos). This led to a poor separation of concerns and, over time, to a growing number of customized environments that users needed to configure and manage separately.</p>
-In Flink 1.10, job submission logic is abstracted into the generic `Executor` interface ([FLIP-73](https://cwiki.apache.org/confluence/display/FLINK/FLIP-73%3A+Introducing+Executors+for+job+submission)). The addition of the `ExecutorCLI` ([FLIP-81](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=133631524)) introduces a unified way to specify configuration parameters for **any** [execution target]({{ site.DOCS_BASE_URL }}flink-docs-release-1.10/ops/cli.html#deployment-ta [...]
+<p>In Flink 1.10, job submission logic is abstracted into the generic <code>Executor</code> interface (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-73%3A+Introducing+Executors+for+job+submission">FLIP-73</a>). The addition of the <code>ExecutorCLI</code> (<a href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=133631524">FLIP-81</a>) introduces a unified way to specify configura [...]
-<span>
+<p><span>
<center>
- <img vspace="8" style="width:100%" src="{{site.baseurl}}/img/blog/2020-02-11-release-1.10.0/flink_1.10_zeppelin.png" />
+ <img vspace="8" style="width:100%" src="/img/blog/2020-02-11-release-1.10.0/flink_1.10_zeppelin.png" />
</center>
-</span>
+</span></p>
-In particular, these changes make it much easier to programmatically use Flink in downstream frameworks — for example, Apache Beam or Zeppelin interactive notebooks — by providing users with a unified entry point to Flink. For users working with Flink across multiple target environments, the transition to a configuration-based execution process also significantly reduces boilerplate code and maintainability overhead.
+<p>In particular, these changes make it much easier to programmatically use Flink in downstream frameworks — for example, Apache Beam or Zeppelin interactive notebooks — by providing users with a unified entry point to Flink. For users working with Flink across multiple target environments, the transition to a configuration-based execution process also significantly reduces boilerplate code and maintainability overhead.</p>
-### Native Kubernetes Integration (Beta)
+<h3 id="native-kubernetes-integration-beta">Native Kubernetes Integration (Beta)</h3>
-For users looking to get started with Flink on a containerized environment, deploying and managing a standalone cluster on top of Kubernetes requires some upfront knowledge about containers, operators and environment-specific tools like `kubectl`.
+<p>For users looking to get started with Flink on a containerized environment, deploying and managing a standalone cluster on top of Kubernetes requires some upfront knowledge about containers, operators and environment-specific tools like <code>kubectl</code>.</p>
-In Flink 1.10, we rolled out the first phase of **Active Kubernetes Integration** ([FLINK-9953](https://jira.apache.org/jira/browse/FLINK-9953)) with support for session clusters (with per-job planned). In this context, “active” means that Flink’s ResourceManager (`K8sResMngr`) natively communicates with Kubernetes to allocate new pods on-demand, similar to Flink’s Yarn and Mesos integration. Users can also leverage namespaces to launch Flink clusters for multi-tenant environments with l [...]
+<p>In Flink 1.10, we rolled out the first phase of <strong>Active Kubernetes Integration</strong> (<a href="https://jira.apache.org/jira/browse/FLINK-9953">FLINK-9953</a>) with support for session clusters (with per-job planned). In this context, “active” means that Flink’s ResourceManager (<code>K8sResMngr</code>) natively communicates with Kubernetes to allocate new pods on-demand, similar to Flink’s Yarn and Mesos integration. Users [...]
-<span>
+<p><span>
<center>
- <img vspace="8" style="width:75%" src="{{site.baseurl}}/img/blog/2020-02-11-release-1.10.0/flink_1.10_nativek8s.png"/>
+ <img vspace="8" style="width:75%" src="/img/blog/2020-02-11-release-1.10.0/flink_1.10_nativek8s.png" />
</center>
-</span>
+</span></p>
-As introduced in [Unified Logic For Job Submission](#unified-logic-for-job-submission), all command-line options in Flink 1.10 are mapped to a unified configuration. For this reason, users can simply refer to the Kubernetes config options and submit a job to an existing Flink session on Kubernetes in the CLI using:
+<p>As introduced in <a href="#unified-logic-for-job-submission">Unified Logic For Job Submission</a>, all command-line options in Flink 1.10 are mapped to a unified configuration. For this reason, users can simply refer to the Kubernetes config options and submit a job to an existing Flink session on Kubernetes in the CLI using:</p>
-```bash
-./bin/flink run -d -e kubernetes-session -Dkubernetes.cluster-id=<ClusterId> examples/streaming/WindowJoin.jar
-```
+<div class="highlight"><pre><code class="language-bash">./bin/flink run -d -e kubernetes-session -Dkubernetes.cluster-id<span class="o">=</span>&lt;ClusterId&gt; examples/streaming/WindowJoin.jar</code></pre></div>
-If you want to try out this preview feature, we encourage you to walk through the [Native Kubernetes setup]({{ site.DOCS_BASE_URL }}flink-docs-release-1.10/ops/deployment/native_kubernetes.html), play around with it and share feedback with the community.
+<p>If you want to try out this preview feature, we encourage you to walk through the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/deployment/native_kubernetes.html">Native Kubernetes setup</a>, play around with it and share feedback with the community.</p>
-### Table API/SQL: Production-ready Hive Integration
+<h3 id="table-apisql-production-ready-hive-integration">Table API/SQL: Production-ready Hive Integration</h3>
-Hive integration was announced as a preview feature in Flink 1.9. This preview allowed users to persist Flink-specific metadata (e.g. Kafka tables) in Hive Metastore using SQL DDL, call UDFs defined in Hive and use Flink for reading and writing Hive tables. Flink 1.10 rounds up this effort with further developments that bring production-ready Hive integration to Flink with full compatibility of [most Hive versions]({{ site.DOCS_BASE_URL }}flink-docs-release-1.10/dev/table/hive/#supported [...]
+<p>Hive integration was announced as a preview feature in Flink 1.9. This preview allowed users to persist Flink-specific metadata (e.g. Kafka tables) in Hive Metastore using SQL DDL, call UDFs defined in Hive and use Flink for reading and writing Hive tables. Flink 1.10 rounds up this effort with further developments that bring production-ready Hive integration to Flink with full compatibility of <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/tab [...]
-#### Native Partition Support for Batch SQL
+<h4 id="native-partition-support-for-batch-sql">Native Partition Support for Batch SQL</h4>
-So far, only writes to non-partitioned Hive tables were supported. In Flink 1.10, the Flink SQL syntax has been extended with `INSERT OVERWRITE` and `PARTITION` ([FLIP-63](https://cwiki.apache.org/confluence/display/FLINK/FLIP-63%3A+Rework+table+partition+support)), enabling users to write into both static and dynamic partitions in Hive.
+<p>So far, only writes to non-partitioned Hive tables were supported. In Flink 1.10, the Flink SQL syntax has been extended with <code>INSERT OVERWRITE</code> and <code>PARTITION</code> (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-63%3A+Rework+table+partition+support">FLIP-63</a>), enabling users to write into both static and dynamic partitions in Hive.</p>
-**Static Partition Writing**
+<p><strong>Static Partition Writing</strong></p>
-```sql
-INSERT { INTO | OVERWRITE } TABLE tablename1 [PARTITION (partcol1=val1, partcol2=val2 ...)] select_statement1 FROM from_statement;
-```
+<div class="highlight"><pre><code class="language-sql"><span class="k">INSERT</span> <span class="err">{</span> <span class="k">INTO</span> <span class="o">|</span> <span class="n">OVERWRITE</span> <span class="err">}</span> <span class="k">TABLE</span> <span class="n">tablename1< [...]
-**Dynamic Partition Writing**
+<p><strong>Dynamic Partition Writing</strong></p>
-```sql
-INSERT { INTO | OVERWRITE } TABLE tablename1 select_statement1 FROM from_statement;
-```
+<div class="highlight"><pre><code class="language-sql"><span class="k">INSERT</span> <span class="err">{</span> <span class="k">INTO</span> <span class="o">|</span> <span class="n">OVERWRITE</span> <span class="err">}</span> <span class="k">TABLE</span> <span class="n">tablename1< [...]
-Fully supporting partitioned tables allows users to take advantage of partition pruning on read, which significantly increases the performance of these operations by reducing the amount of data that needs to be scanned.
+<p>Fully supporting partitioned tables allows users to take advantage of partition pruning on read, which significantly increases the performance of these operations by reducing the amount of data that needs to be scanned.</p>
-#### Further Optimizations
+<h4 id="further-optimizations">Further Optimizations</h4>
-Besides partition pruning, Flink 1.10 introduces more [read optimizations]({{ site.DOCS_BASE_URL }}flink-docs-release-1.10/dev/table/hive/read_write_hive.html#optimizations) to Hive integration, such as:
+<p>Besides partition pruning, Flink 1.10 introduces more <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/hive/read_write_hive.html#optimizations">read optimizations</a> to Hive integration, such as:</p>
-* **Projection pushdown:** Flink leverages projection pushdown to minimize data transfer between Flink and Hive tables by omitting unnecessary fields from table scans. This is especially beneficial for tables with a large number of columns.
-
-* **LIMIT pushdown:** for queries with the `LIMIT` clause, Flink will limit the number of output records wherever possible to minimize the amount of data transferred across the network.
-
-* **ORC Vectorization on Read:** to boost read performance for ORC files, Flink now uses the native ORC Vectorized Reader by default for Hive versions above 2.0.0 and columns with non-complex data types.
+<ul>
+ <li>
+ <p><strong>Projection pushdown:</strong> Flink leverages projection pushdown to minimize data transfer between Flink and Hive tables by omitting unnecessary fields from table scans. This is especially beneficial for tables with a large number of columns.</p>
+ </li>
+ <li>
+ <p><strong>LIMIT pushdown:</strong> for queries with the <code>LIMIT</code> clause, Flink will limit the number of output records wherever possible to minimize the amount of data transferred across the network.</p>
+ </li>
+ <li>
+ <p><strong>ORC Vectorization on Read:</strong> to boost read performance for ORC files, Flink now uses the native ORC Vectorized Reader by default for Hive versions above 2.0.0 and columns with non-complex data types.</p>
+ </li>
+</ul>
-#### Pluggable Modules as Flink System Objects (Beta)
+<h4 id="pluggable-modules-as-flink-system-objects-beta">Pluggable Modules as Flink System Objects (Beta)</h4>
-Flink 1.10 introduces a generic mechanism for pluggable modules in the Flink table core, with a first focus on system functions ([FLIP-68](https://cwiki.apache.org/confluence/display/FLINK/FLIP-68%3A+Extend+Core+Table+System+with+Pluggable+Modules)). With modules, users can extend Flink’s system objects — for example use Hive built-in functions that behave like Flink system functions. This release ships with a pre-implemented `HiveModule`, supporting multiple Hive versions, but users are [...]
+<p>Flink 1.10 introduces a generic mechanism for pluggable modules in the Flink table core, with a first focus on system functions (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-68%3A+Extend+Core+Table+System+with+Pluggable+Modules">FLIP-68</a>). With modules, users can extend Flink’s system objects — for example use Hive built-in functions that behave like Flink system functions. This release ships with a pre-implemented <code>HiveModu [...]
-### Other Improvements to the Table API/SQL
+<h3 id="other-improvements-to-the-table-apisql">Other Improvements to the Table API/SQL</h3>
-#### Watermarks and Computed Columns in SQL DDL
+<h4 id="watermarks-and-computed-columns-in-sql-ddl">Watermarks and Computed Columns in SQL DDL</h4>
-Flink 1.10 supports stream-specific syntax extensions to define time attributes and watermark generation in Flink SQL DDL ([FLIP-66](https://cwiki.apache.org/confluence/display/FLINK/FLIP-66%3A+Support+Time+Attribute+in+SQL+DDL)). This allows time-based operations, like windowing, and the definition of [watermark strategies]({{ site.DOCS_BASE_URL }}flink-docs-release-1.10/dev/table/sql/create.html#create-table) on tables created using DDL statements.
+<p>Flink 1.10 supports stream-specific syntax extensions to define time attributes and watermark generation in Flink SQL DDL (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-66%3A+Support+Time+Attribute+in+SQL+DDL">FLIP-66</a>). This allows time-based operations, like windowing, and the definition of <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/sql/create.html#create-table">watermark strategies [...]
-```sql
-CREATE TABLE table_name (
+<div class="highlight"><pre><code class="language-sql"><span class="k">CREATE</span> <span class="k">TABLE</span> <span class="k">table_name</span> <span class="p">(</span>
- WATERMARK FOR columnName AS <watermark_strategy_expression>
+ <span class="n">WATERMARK</span> <span class="k">FOR</span> <span class="n">columnName</span> <span class="k">AS</span> <span class="o">&lt;</span><span class="n">watermark_strategy_expression</span><span class="o">&gt;</span>
-) WITH (
- ...
-)
-```
+<span class="p">)</span> <span class="k">WITH</span> <span class="p">(</span>
+ <span class="p">...</span>
+<span class="p">)</span></code></pre></div>
-This release also introduces support for virtual computed columns ([FLIP-70](https://cwiki.apache.org/confluence/display/FLINK/FLIP-70%3A+Flink+SQL+Computed+Column+Design)) that can be derived based on other columns in the same table or deterministic expressions (i.e. literal values, UDFs and built-in functions). In Flink, computed columns are useful to define time attributes [upon table creation]({{ site.DOCS_BASE_URL }}flink-docs-release-1.10/dev/table/sql/create.html#create-table).
+<p>This release also introduces support for virtual computed columns (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-70%3A+Flink+SQL+Computed+Column+Design">FLIP-70</a>) that can be derived based on other columns in the same table or deterministic expressions (i.e. literal values, UDFs and built-in functions). In Flink, computed columns are useful to define time attributes <a href="https://ci.apache.org/projects/flink/flink-docs-releas [...]
-#### Additional Extensions to SQL DDL
+<h4 id="additional-extensions-to-sql-ddl">Additional Extensions to SQL DDL</h4>
-There is now a clear distinction between temporary/persistent and system/catalog functions ([FLIP-57](https://cwiki.apache.org/confluence/display/FLINK/FLIP-57%3A+Rework+FunctionCatalog)). This not only eliminates ambiguity in function reference, but also allows for deterministic function resolution order (i.e. in case of naming collision, system functions will precede catalog functions, with temporary functions taking precedence over persistent functions for both dimensions).
+<p>There is now a clear distinction between temporary/persistent and system/catalog functions (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-57%3A+Rework+FunctionCatalog">FLIP-57</a>). This not only eliminates ambiguity in function reference, but also allows for deterministic function resolution order (i.e. in case of naming collision, system functions will precede catalog functions, with temporary functions taking precedence over persistent [...]
-Following the groundwork in FLIP-57, we extended the SQL DDL syntax to support the creation of catalog functions, temporary functions and temporary system functions ([FLIP-79](https://cwiki.apache.org/confluence/display/FLINK/FLIP-79+Flink+Function+DDL+Support)):
+<p>Following the groundwork in FLIP-57, we extended the SQL DDL syntax to support the creation of catalog functions, temporary functions and temporary system functions (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-79+Flink+Function+DDL+Support">FLIP-79</a>):</p>
-```sql
-CREATE [TEMPORARY|TEMPORARY SYSTEM] FUNCTION
+<div class="highlight"><pre><code class="language-sql"><span class="k">CREATE</span> <span class="p">[</span><span class="k">TEMPORARY</span><span class="o">|</span><span class="k">TEMPORARY</span> <span class="k">SYSTEM</span><span class="p">]</span> <span class="k">FUNCTION</span>
- [IF NOT EXISTS] [catalog_name.][db_name.]function_name
+ <span class="p">[</span><span class="n">IF</span> <span class="k">NOT</span> <span class="k">EXISTS</span><span class="p">]</span> <span class="p">[</span><span class="k">catalog_name</span><span class="p">.][</span><span class="n">db_name</span><span class="p">.]</span> [...]
-AS identifier [LANGUAGE JAVA|SCALA]
-```
+<span class="k">AS</span> <span class="n">identifier</span> <span class="p">[</span><span class="k">LANGUAGE</span> <span class="n">JAVA</span><span class="o">|</span><span class="n">SCALA</span><span class="p">]</span></code></pre></div>
-For a complete overview of the current state of DDL support in Flink SQL, check the [updated documentation]({{ site.DOCS_BASE_URL }}flink-docs-release-1.10/dev/table/sql/).
+<p>For a complete overview of the current state of DDL support in Flink SQL, check the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/sql/">updated documentation</a>.</p>
-<span class="label label-danger">Note</span> In order to correctly handle and guarantee a consistent behavior across meta-objects (tables, views, functions) in the future, some object declaration methods in the Table API have been deprecated in favor of methods that are closer to standard SQL DDL ([FLIP-64](https://cwiki.apache.org/confluence/display/FLINK/FLIP-64%3A+Support+for+Temporary+Objects+in+Table+module)).
+<p><span class="label label-danger">Note</span> In order to correctly handle and guarantee a consistent behavior across meta-objects (tables, views, functions) in the future, some object declaration methods in the Table API have been deprecated in favor of methods that are closer to standard SQL DDL (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-64%3A+Support+for+Temporary+Objects+in+Table+module">FLIP-64</a>).</p>
-#### Full TPC-DS Coverage for Batch
+<h4 id="full-tpc-ds-coverage-for-batch">Full TPC-DS Coverage for Batch</h4>
-TPC-DS is a widely used industry-standard decision support benchmark to evaluate and measure the performance of SQL-based data processing engines. In Flink 1.10, all TPC-DS queries are supported end-to-end ([FLINK-11491](https://issues.apache.org/jira/browse/FLINK-11491)), reflecting the readiness of its SQL engine to address the needs of modern data warehouse-like workloads.
+<p>TPC-DS is a widely used industry-standard decision support benchmark to evaluate and measure the performance of SQL-based data processing engines. In Flink 1.10, all TPC-DS queries are supported end-to-end (<a href="https://issues.apache.org/jira/browse/FLINK-11491">FLINK-11491</a>), reflecting the readiness of its SQL engine to address the needs of modern data warehouse-like workloads.</p>
-### PyFlink: Support for Native User Defined Functions (UDFs)
+<h3 id="pyflink-support-for-native-user-defined-functions-udfs">PyFlink: Support for Native User Defined Functions (UDFs)</h3>
-A preview of PyFlink was introduced in the previous release, making headway towards the goal of full Python support in Flink. For this release, the focus was to enable users to register and use Python User-Defined Functions (UDF, with UDTF/UDAF planned) in the Table API/SQL ([FLIP-58](https://cwiki.apache.org/confluence/display/FLINK/FLIP-58%3A+Flink+Python+User-Defined+Stateless+Function+for+Table)).
+<p>A preview of PyFlink was introduced in the previous release, making headway towards the goal of full Python support in Flink. For this release, the focus was to enable users to register and use Python User-Defined Functions (UDF, with UDTF/UDAF planned) in the Table API/SQL (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-58%3A+Flink+Python+User-Defined+Stateless+Function+for+Table">FLIP-58</a>).</p>
-<span>
+<p><span>
<center>
- <img vspace="8" hspace="100" style="width:75%" src="{{site.baseurl}}/img/blog/2020-02-11-release-1.10.0/flink_1.10_pyflink.gif"/>
+ <img vspace="8" hspace="100" style="width:75%" src="/img/blog/2020-02-11-release-1.10.0/flink_1.10_pyflink.gif" />
</center>
-</span>
-
-If you are interested in the underlying implementation — leveraging Apache Beam’s [Portability Framework](https://beam.apache.org/roadmap/portability/) — refer to the “Architecture” section of FLIP-58 and also to [FLIP-78](https://cwiki.apache.org/confluence/display/FLINK/FLIP-78%3A+Flink+Python+UDF+Environment+and+Dependency+Management). These data structures lay the required foundation for Pandas support and for PyFlink to eventually reach the DataStream API.
-
-From Flink 1.10, users can also easily install PyFlink through `pip` using:
-
-```bash
-pip install apache-flink
-```
-
-For a preview of other improvements planned for PyFlink, check [FLINK-14500](https://issues.apache.org/jira/browse/FLINK-14500) and get involved in the [discussion](http://apache-flink.147419.n8.nabble.com/Re-DISCUSS-What-parts-of-the-Python-API-should-we-focus-on-next-td1285.html) for requested user features.
-
-## Important Changes
-
- * [[FLINK-10725](https://issues.apache.org/jira/browse/FLINK-10725)] Flink can now be compiled and run on Java 11.
-
- * [[FLINK-15495](https://jira.apache.org/jira/browse/FLINK-15495)] The Blink planner is now the default in the SQL Client, so that users can benefit from all the latest features and improvements. The switch from the old planner in the Table API is also planned for the next release, so we recommend that users start getting familiar with the Blink planner.
-
- * [[FLINK-13025](https://issues.apache.org/jira/browse/FLINK-13025)] There is a [new Elasticsearch sink connector](https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/connectors/elasticsearch.html#elasticsearch-connector), fully supporting Elasticsearch 7.x versions.
+</span></p>
- * [[FLINK-15115](https://issues.apache.org/jira/browse/FLINK-15115)] The connectors for Kafka 0.8 and 0.9 have been marked as deprecated and will no longer be actively supported. If you are still using these versions or have any other related concerns, please reach out to the @dev mailing list.
+<p>If you are interested in the underlying implementation — leveraging Apache Beam’s <a href="https://beam.apache.org/roadmap/portability/">Portability Framework</a> — refer to the “Architecture” section of FLIP-58 and also to <a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-78%3A+Flink+Python+UDF+Environment+and+Dependency+Management">FLIP-78</a>. These data structures lay the required foundation for Pandas support and for [...]
- * [[FLINK-14516](https://issues.apache.org/jira/browse/FLINK-14516)] The non-credit-based network flow control code was removed, along with the configuration option `taskmanager.network.credit.model`. Moving forward, Flink will always use credit-based flow control.
+<p>From Flink 1.10, users can also easily install PyFlink through <code>pip</code> using:</p>
- * [[FLINK-12122](https://issues.apache.org/jira/browse/FLINK-12122)] [FLIP-6](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65147077) was rolled out with Flink 1.5.0 and introduced a code regression related to the way slots are allocated from `TaskManagers`. To use a scheduling strategy that is closer to the pre-FLIP behavior, where Flink tries to spread out the workload across all currently available `TaskManagers`, users can set `cluster.evenly-spread-out-slots: tru [...]
+<div class="highlight"><pre><code class="language-bash">pip install apache-flink</code></pre></div>
- * [[FLINK-11956](https://issues.apache.org/jira/browse/FLINK-11956)] `s3-hadoop` and `s3-presto` filesystems no longer use class relocations and should be loaded through [plugins]({{ site.DOCS_BASE_URL }}flink-docs-release-1.10/ops/filesystems/#pluggable-file-systems), but now seamlessly integrate with all credential providers. Other filesystems are strongly recommended to be used only as plugins, as we will continue to remove relocations.
+<p>For a preview of other improvements planned for PyFlink, check <a href="https://issues.apache.org/jira/browse/FLINK-14500">FLINK-14500</a> and get involved in the <a href="http://apache-flink.147419.n8.nabble.com/Re-DISCUSS-What-parts-of-the-Python-API-should-we-focus-on-next-td1285.html">discussion</a> for requested user features.</p>
- * Flink 1.9 shipped with a refactored Web UI, with the legacy one being kept around as backup in case something wasn’t working as expected. No issues have been reported so far, so [the community voted](http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Remove-old-WebUI-td35218.html) to drop the legacy Web UI in Flink 1.10.
+<h2 id="important-changes">Important Changes</h2>
+<ul>
+ <li>
+ <p>[<a href="https://issues.apache.org/jira/browse/FLINK-10725">FLINK-10725</a>] Flink can now be compiled and run on Java 11.</p>
+ </li>
+ <li>
+ <p>[<a href="https://jira.apache.org/jira/browse/FLINK-15495">FLINK-15495</a>] The Blink planner is now the default in the SQL Client, so that users can benefit from all the latest features and improvements. The switch from the old planner in the Table API is also planned for the next release, so we recommend that users start getting familiar with the Blink planner.</p>
+ </li>
+ <li>
+ <p>[<a href="https://issues.apache.org/jira/browse/FLINK-13025">FLINK-13025</a>] There is a <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/connectors/elasticsearch.html#elasticsearch-connector">new Elasticsearch sink connector</a>, fully supporting Elasticsearch 7.x versions.</p>
+ </li>
+ <li>
+ <p>[<a href="https://issues.apache.org/jira/browse/FLINK-15115">FLINK-15115</a>] The connectors for Kafka 0.8 and 0.9 have been marked as deprecated and will no longer be actively supported. If you are still using these versions or have any other related concerns, please reach out to the @dev mailing list.</p>
+ </li>
+ <li>
+ <p>[<a href="https://issues.apache.org/jira/browse/FLINK-14516">FLINK-14516</a>] The non-credit-based network flow control code was removed, along with the configuration option <code>taskmanager.network.credit.model</code>. Moving forward, Flink will always use credit-based flow control.</p>
+ </li>
+ <li>
+ <p>[<a href="https://issues.apache.org/jira/browse/FLINK-12122">FLINK-12122</a>] <a href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65147077">FLIP-6</a> was rolled out with Flink 1.5.0 and introduced a code regression related to the way slots are allocated from <code>TaskManagers</code>. To use a scheduling strategy that is closer to the pre-FLIP behavior, where Flink tries to spread out the workload [...]
+ </li>
+ <li>
+ <p>[<a href="https://issues.apache.org/jira/browse/FLINK-11956">FLINK-11956</a>] <code>s3-hadoop</code> and <code>s3-presto</code> filesystems no longer use class relocations and should be loaded through <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/filesystems/#pluggable-file-systems">plugins</a>, but now seamlessly integrate with all credential providers. Other filesystems are stro [...]
+ </li>
+ <li>
+ <p>Flink 1.9 shipped with a refactored Web UI, with the legacy one being kept around as backup in case something wasn’t working as expected. No issues have been reported so far, so <a href="http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Remove-old-WebUI-td35218.html">the community voted</a> to drop the legacy Web UI in Flink 1.10.</p>
+ </li>
+</ul>
-## Release Notes
-
-Please review the [release notes]({{ site.DOCS_BASE_URL }}flink-docs-release-1.10/release-notes/flink-1.10.html) carefully for a detailed list of changes and new features if you plan to upgrade your setup to Flink 1.10. This version is API-compatible with previous 1.x releases for APIs annotated with the @Public annotation.
+<h2 id="release-notes">Release Notes</h2>
+<p>Please review the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/release-notes/flink-1.10.html">release notes</a> carefully for a detailed list of changes and new features if you plan to upgrade your setup to Flink 1.10. This version is API-compatible with previous 1.x releases for APIs annotated with the @Public annotation.</p>
-## List of Contributors
+<h2 id="list-of-contributors">List of Contributors</h2>
-The Apache Flink community would like to thank all contributors that have made this release possible:
+<p>The Apache Flink community would like to thank all contributors that have made this release possible:</p>
-Achyuth Samudrala, Aitozi, Alberto Romero, Alec.Ch, Aleksey Pak, Alexander Fedulov, Alice Yan, Aljoscha Krettek, Aloys, Andrey Zagrebin, Arvid Heise, Benchao Li, Benoit Hanotte, Benoît Paris, Bhagavan Das, Biao Liu, Chesnay Schepler, Congxian Qiu, Cyrille Chépélov, César Soto Valero, David Anderson, David Hrbacek, David Moravek, Dawid Wysakowicz, Dezhi Cai, Dian Fu, Dyana Rose, Eamon Taaffe, Fabian Hueske, Fawad Halim, Fokko Driesprong, Frey Gao, Gabor Gevay, Gao Yun, Gary Yao, GatsbyNew [...]
+<p>Achyuth Samudrala, Aitozi, Alberto Romero, Alec.Ch, Aleksey Pak, Alexander Fedulov, Alice Yan, Aljoscha Krettek, Aloys, Andrey Zagrebin, Arvid Heise, Benchao Li, Benoit Hanotte, Benoît Paris, Bhagavan Das, Biao Liu, Chesnay Schepler, Congxian Qiu, Cyrille Chépélov, César Soto Valero, David Anderson, David Hrbacek, David Moravek, Dawid Wysakowicz, Dezhi Cai, Dian Fu, Dyana Rose, Eamon Taaffe, Fabian Hueske, Fawad Halim, Fokko Driesprong, Frey Gao, Gabor Gevay, Gao Yun, Gary Yao, [...]
</description>
-<pubDate>Tue, 11 Feb 2020 02:30:00 +0000</pubDate>
+<pubDate>Tue, 11 Feb 2020 03:30:00 +0100</pubDate>
<link>https://flink.apache.org/news/2020/02/11/release-1.10.0.html</link>
<guid isPermaLink="true">/news/2020/02/11/release-1.10.0.html</guid>
</item>
<item>
<title>A Guide for Unit Testing in Apache Flink</title>
-<description>Writing unit tests is one of the essential tasks of designing a production-grade application. Without tests, a single change in code can result in cascades of failure in production. Thus unit tests should be written for all types of applications, be it a simple job cleaning data and training a model or a complex multi-tenant, real-time data processing system. In the following sections, we provide a guide for unit testing of Apache Flink applications.
-Apache Flink provides a robust unit testing framework to make sure your applications behave in production as expected during development. You need to include the following dependencies to utilize the provided framework.
-
-```xml
-<dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-test-utils_${scala.binary.version}</artifactId>
- <version>${flink.version}</version>
- <scope>test</scope>
-</dependency>
-<dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-runtime_2.11</artifactId>
- <version>1.9.0</version>
- <scope>test</scope>
- <classifier>tests</classifier>
-</dependency>
-<dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-streaming-java_2.11</artifactId>
- <version>1.9.0</version>
- <scope>test</scope>
- <classifier>tests</classifier>
-</dependency>
-```
-
-The strategy of writing unit tests differs for various operators. You can break down the strategy into the following three buckets:
-
-* Stateless Operators
-* Stateful Operators
-* Timed Process Operators
-
-
-# Stateless Operators
-
-Writing unit tests for a stateless operator is a breeze. You need to follow the basic norm of writing a test case, i.e., create an instance of the function class and test the appropriate methods. Let’s take an example of a simple `Map` operator.
-
-```java
-public class MyStatelessMap implements MapFunction<String, String> {
- @Override
- public String map(String in) throws Exception {
- String out = "hello " + in;
- return out;
- }
-}
-```
-
-The test case for the above operator should look like
-
-```java
-@Test
-public void testMap() throws Exception {
- MyStatelessMap statelessMap = new MyStatelessMap();
- String out = statelessMap.map("world");
- Assert.assertEquals("hello world", out);
-}
-```
-
-Pretty simple, right? Let’s take a look at one for the `FlatMap` operator.
-
-```java
-public class MyStatelessFlatMap implements FlatMapFunction<String, String> {
- @Override
- public void flatMap(String in, Collector<String> collector) throws Exception {
- String out = "hello " + in;
- collector.collect(out);
- }
-}
-```
-
-`FlatMap` operators require a `Collector` object along with the input. For the test case, we have two options:
-
-1. Mock the `Collector` object using Mockito
-2. Use the `ListCollector` provided by Flink
-
-I prefer the second method as it requires fewer lines of code and is suitable for most of the cases.
-
-```java
-@Test
-public void testFlatMap() throws Exception {
- MyStatelessFlatMap statelessFlatMap = new MyStatelessFlatMap();
- List<String> out = new ArrayList<>();
- ListCollector<String> listCollector = new ListCollector<>(out);
- statelessFlatMap.flatMap("world", listCollector);
- Assert.assertEquals(Lists.newArrayList("hello world"), out);
-}
-```
-
-
-# Stateful Operators
-
-Writing test cases for stateful operators requires more effort. You need to check whether the operator state is updated correctly and if it is cleaned up properly along with the output of the operator.
-
-Let’s take an example of stateful `FlatMap` function
-
-```java
-public class StatefulFlatMap extends RichFlatMapFunction<String, String> {
- ValueState<String> previousInput;
-
- @Override
- public void open(Configuration parameters) throws Exception {
- previousInput = getRuntimeContext().getState(
- new ValueStateDescriptor<String>("previousInput", Types.STRING));
- }
-
- @Override
- public void flatMap(String in, Collector<String> collector) throws Exception {
- String out = "hello " + in;
- if(previousInput.value() != null){
- out = out + " " + previousInput.value();
- }
- previousInput.update(in);
- collector.collect(out);
- }
-}
-```
-
-The intricate part of writing tests for the above class is to mock the configuration as well as the runtime context of the application. Flink provides TestHarness classes so that users don’t have to create the mock objects themselves. Using the `KeyedOperatorHarness`, the test looks like:
-
-```java
-import org.apache.flink.streaming.api.operators.StreamFlatMap;
-import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
-import org.apache.flink.streaming.util.KeyedOneInputStreamOperatorTestHarness;
-import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
-
-@Test
-public void testFlatMap() throws Exception{
- StatefulFlatMap statefulFlatMap = new StatefulFlatMap();
-
- // OneInputStreamOperatorTestHarness takes the input and output types as type parameters
- OneInputStreamOperatorTestHarness<String, String> testHarness =
- // KeyedOneInputStreamOperatorTestHarness takes three arguments:
- // Flink operator object, key selector and key type
- new KeyedOneInputStreamOperatorTestHarness<>(
- new StreamFlatMap<>(statefulFlatMap), x -> "1", Types.STRING);
- testHarness.open();
-
- // test first record
- testHarness.processElement("world", 10);
- ValueState<String> previousInput =
- statefulFlatMap.getRuntimeContext().getState(
- new ValueStateDescriptor<>("previousInput", Types.STRING));
- String stateValue = previousInput.value();
- Assert.assertEquals(
- Lists.newArrayList(new StreamRecord<>("hello world", 10)),
- testHarness.extractOutputStreamRecords());
- Assert.assertEquals("world", stateValue);
-
- // test second record
- testHarness.processElement("parallel", 20);
- Assert.assertEquals(
- Lists.newArrayList(
- new StreamRecord<>("hello world", 10),
- new StreamRecord<>("hello parallel world", 20)),
- testHarness.extractOutputStreamRecords());
- Assert.assertEquals("parallel", previousInput.value());
-}
-```
-
-The test harness provides many helper methods, three of which are being used here:
-
-1. `open`: calls the open of the `FlatMap` function with relevant parameters. It also initializes the context.
-2. `processElement`: allows users to pass an input element as well as the timestamp associated with the element.
-3. `extractOutputStreamRecords`: gets the output records along with their timestamps from the `Collector`.
-
-The test harness simplifies the unit testing for the stateful functions to a large extent.
-
-You might also need to check whether the state value is being set correctly. You can get the state value directly from the operator using a mechanism similar to the one used while creating the state. This is also demonstrated in the previous example.
-
-
-# Timed Process Operators
-
-Writing tests for process functions, that work with time, is quite similar to writing tests for stateful functions because you can also use test harness.
-However, you need to take care of another aspect, which is providing timestamps for events and controlling the current time of the application. By setting the current (processing or event) time, you can trigger registered timers, which will call the `onTimer` method of the function
-
-```java
-public class MyProcessFunction extends KeyedProcessFunction<String, String, String> {
- @Override
- public void processElement(String in, Context context, Collector<String> collector) throws Exception {
- context.timerService().registerProcessingTimeTimer(50);
- String out = "hello " + in;
- collector.collect(out);
- }
-
- @Override
- public void onTimer(long timestamp, OnTimerContext ctx, Collector<String> out) throws Exception {
- out.collect(String.format("Timer triggered at timestamp %d", timestamp));
- }
-}
-```
-
-We need to test both the methods in the `KeyedProcessFunction`, i.e., `processElement` as well as `onTimer`. Using a test harness, we can control the current time of the function. Thus, we can trigger the timer at will rather than waiting for a specific time.
-
-Let’s take a look at the test case
-
-```java
-@Test
-public void testProcessElement() throws Exception{
- MyProcessFunction myProcessFunction = new MyProcessFunction();
- OneInputStreamOperatorTestHarness<String, String> testHarness =
- new KeyedOneInputStreamOperatorTestHarness<>(
- new KeyedProcessOperator<>(myProcessFunction), x -> "1", Types.STRING);
-
- // Function time is initialized to 0
- testHarness.open();
- testHarness.processElement("world", 10);
-
- Assert.assertEquals(
- Lists.newArrayList(new StreamRecord<>("hello world", 10)),
- testHarness.extractOutputStreamRecords());
-}
-
-@Test
-public void testOnTimer() throws Exception {
- MyProcessFunction myProcessFunction = new MyProcessFunction();
- OneInputStreamOperatorTestHarness<String, String> testHarness =
- new KeyedOneInputStreamOperatorTestHarness<>(
- new KeyedProcessOperator<>(myProcessFunction), x -> "1", Types.STRING);
-
- testHarness.open();
- testHarness.processElement("world", 10);
- Assert.assertEquals(1, testHarness.numProcessingTimeTimers());
+<description><p>Writing unit tests is one of the essential tasks of designing a production-grade application. Without tests, a single change in code can result in cascades of failure in production. Thus unit tests should be written for all types of applications, be it a simple job cleaning data and training a model or a complex multi-tenant, real-time data processing system. In the following sections, we provide a guide for unit testing of Apache Flink applications.
+Apache Flink provides a robust unit testing framework to make sure your applications behave in production as expected during development. You need to include the following dependencies to utilize the provided framework.</p>
+
+<div class="highlight"><pre><code class="language-xml"><span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-test-utils_${scala.binary.version}<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>${flink.version}<span class="nt">&lt;/version&gt;</span>
+ <span class="nt">&lt;scope&gt;</span>test<span class="nt">&lt;/scope&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span>
+<span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-runtime_2.11<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.9.0<span class="nt">&lt;/version&gt;</span>
+ <span class="nt">&lt;scope&gt;</span>test<span class="nt">&lt;/scope&gt;</span>
+ <span class="nt">&lt;classifier&gt;</span>tests<span class="nt">&lt;/classifier&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span>
+<span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-streaming-java_2.11<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.9.0<span class="nt">&lt;/version&gt;</span>
+ <span class="nt">&lt;scope&gt;</span>test<span class="nt">&lt;/scope&gt;</span>
+ <span class="nt">&lt;classifier&gt;</span>tests<span class="nt">&lt;/classifier&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span></code></pre></div>
+
+<p>The strategy of writing unit tests differs for various operators. You can break down the strategy into the following three buckets:</p>
+
+<ul>
+ <li>Stateless Operators</li>
+ <li>Stateful Operators</li>
+ <li>Timed Process Operators</li>
+</ul>
+
+<h1 id="stateless-operators">Stateless Operators</h1>
+
+<p>Writing unit tests for a stateless operator is a breeze. You need to follow the basic norm of writing a test case, i.e., create an instance of the function class and test the appropriate methods. Let’s take an example of a simple <code>Map</code> operator.</p>
+
+<div class="highlight"><pre><code class="language-java"><span class="kd">public</span> <span class="kd">class</span> <span class="nc">MyStatelessMap</span> <span class="kd">implements</span> <span class="n">MapFunction</span><span class="o">&lt;</span><span class="n">String</span><span class= [...]
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="n">String</span> <span class="nf">map</span><span class="o">(</span><span class="n">String</span> <span class="n">in</span><span class="o">)</span> <span class="kd">throws</span> <span class="n">Exception</span> <span class="o">{< [...]
+ <span class="n">String</span> <span class="n">out</span> <span class="o">=</span> <span class="s">&quot;hello &quot;</span> <span class="o">+</span> <span class="n">in</span><span class="o">;</span>
+ <span class="k">return</span> <span class="n">out</span><span class="o">;</span>
+ <span class="o">}</span>
+<span class="o">}</span></code></pre></div>
+
+<p>The test case for the above operator should look like</p>
+
+<div class="highlight"><pre><code class="language-java"><span class="nd">@Test</span>
+<span class="kd">public</span> <span class="kt">void</span> <span class="nf">testMap</span><span class="o">()</span> <span class="kd">throws</span> <span class="n">Exception</span> <span class="o">{</span>
+ <span class="n">MyStatelessMap</span> <span class="n">statelessMap</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">MyStatelessMap</span><span class="o">();</span>
+ <span class="n">String</span> <span class="n">out</span> <span class="o">=</span> <span class="n">statelessMap</span><span class="o">.</span><span class="na">map</span><span class="o">(</span><span class="s">&quot;world&quot;</span><span class="o">);</span>
+ <span class="n">Assert</span><span class="o">.</span><span class="na">assertEquals</span><span class="o">(</span><span class="s">&quot;hello world&quot;</span><span class="o">,</span> <span class="n">out</span><span class="o">);</span>
+<span class="o">}</span></code></pre></div>
+
+<p>Pretty simple, right? Let’s take a look at one for the <code>FlatMap</code> operator.</p>
+
+<div class="highlight"><pre><code class="language-java"><span class="kd">public</span> <span class="kd">class</span> <span class="nc">MyStatelessFlatMap</span> <span class="kd">implements</span> <span class="n">FlatMapFunction</span><span class="o">&lt;</span><span class="n">String</span><spa [...]
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="nf">flatMap</span><span class="o">(</span><span class="n">String</span> <span class="n">in</span><span class="o">,</span> <span class="n">Collector</span><span class="o">&lt;</span><span class="n">St [...]
+ <span class="n">String</span> <span class="n">out</span> <span class="o">=</span> <span class="s">&quot;hello &quot;</span> <span class="o">+</span> <span class="n">in</span><span class="o">;</span>
+ <span class="n">collector</span><span class="o">.</span><span class="na">collect</span><span class="o">(</span><span class="n">out</span><span class="o">);</span>
+ <span class="o">}</span>
+<span class="o">}</span></code></pre></div>
+
+<p><code>FlatMap</code> operators require a <code>Collector</code> object along with the input. For the test case, we have two options:</p>
+
+<ol>
+ <li>Mock the <code>Collector</code> object using Mockito</li>
+ <li>Use the <code>ListCollector</code> provided by Flink</li>
+</ol>
+
+<p>I prefer the second method as it requires fewer lines of code and is suitable for most of the cases.</p>
+
+<div class="highlight"><pre><code class="language-java"><span class="nd">@Test</span>
+<span class="kd">public</span> <span class="kt">void</span> <span class="nf">testFlatMap</span><span class="o">()</span> <span class="kd">throws</span> <span class="n">Exception</span> <span class="o">{</span>
+ <span class="n">MyStatelessFlatMap</span> <span class="n">statelessFlatMap</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">MyStatelessFlatMap</span><span class="o">();</span>
+ <span class="n">List</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">out</span> <span class="o">=</span> <span class="k">new</span> <span class="n">ArrayList</span><span class="o">&lt;&gt;();</span>
+ <span class="n">ListCollector</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">listCollector</span> <span class="o">=</span> <span class="k">new</span> <span class="n">ListCollector</span><span class="o">&lt;&gt;(</span> [...]
+ <span class="n">statelessFlatMap</span><span class="o">.</span><span class="na">flatMap</span><span class="o">(</span><span class="s">&quot;world&quot;</span><span class="o">,</span> <span class="n">listCollector</span><span class="o">);</span>
+ <span class="n">Assert</span><span class="o">.</span><span class="na">assertEquals</span><span class="o">(</span><span class="n">Lists</span><span class="o">.</span><span class="na">newArrayList</span><span class="o">(</span><span class="s">&quot;hello world&quot;</span><span clas [...]
+<span class="o">}</span></code></pre></div>
+
+<h1 id="stateful-operators">Stateful Operators</h1>
+
+<p>Writing test cases for stateful operators requires more effort. You need to check whether the operator state is updated correctly and if it is cleaned up properly along with the output of the operator.</p>
+
+<p>Let’s take an example of stateful <code>FlatMap</code> function</p>
+
+<div class="highlight"><pre><code class="language-java"><span class="kd">public</span> <span class="kd">class</span> <span class="nc">StatefulFlatMap</span> <span class="kd">extends</span> <span class="n">RichFlatMapFunction</span><span class="o">&lt;</span><span class="n">String</span><span [...]
+ <span class="n">ValueState</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">previousInput</span><span class="o">;</span>
+
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="nf">open</span><span class="o">(</span><span class="n">Configuration</span> <span class="n">parameters</span><span class="o">)</span> <span class="kd">throws</span> <span class="n">Exception</span> <span class=" [...]
+ <span class="n">previousInput</span> <span class="o">=</span> <span class="n">getRuntimeContext</span><span class="o">().</span><span class="na">getState</span><span class="o">(</span>
+ <span class="k">new</span> <span class="n">ValueStateDescriptor</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;(</span><span class="s">&quot;previousInput&quot;</span><span class="o">,</span> <span class="n">Types</span><span class="o">.</s [...]
+ <span class="o">}</span>
+
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="nf">flatMap</span><span class="o">(</span><span class="n">String</span> <span class="n">in</span><span class="o">,</span> <span class="n">Collector</span><span class="o">&lt;</span><span class="n">St [...]
+ <span class="n">String</span> <span class="n">out</span> <span class="o">=</span> <span class="s">&quot;hello &quot;</span> <span class="o">+</span> <span class="n">in</span><span class="o">;</span>
+ <span class="k">if</span><span class="o">(</span><span class="n">previousInput</span><span class="o">.</span><span class="na">value</span><span class="o">()</span> <span class="o">!=</span> <span class="kc">null</span><span class="o">){</span>
+ <span class="n">out</span> <span class="o">=</span> <span class="n">out</span> <span class="o">+</span> <span class="s">&quot; &quot;</span> <span class="o">+</span> <span class="n">previousInput</span><span class="o">.</span><span class="na">value</span><span class="o&qu [...]
+ <span class="o">}</span>
+ <span class="n">previousInput</span><span class="o">.</span><span class="na">update</span><span class="o">(</span><span class="n">in</span><span class="o">);</span>
+ <span class="n">collector</span><span class="o">.</span><span class="na">collect</span><span class="o">(</span><span class="n">out</span><span class="o">);</span>
+ <span class="o">}</span>
+<span class="o">}</span></code></pre></div>
+
+<p>The intricate part of writing tests for the above class is to mock the configuration as well as the runtime context of the application. Flink provides TestHarness classes so that users don’t have to create the mock objects themselves. Using the <code>KeyedOperatorHarness</code>, the test looks like:</p>
+
+<div class="highlight"><pre><code class="language-java"><span class="kn">import</span> <span class="nn">org.apache.flink.streaming.api.operators.StreamFlatMap</span><span class="o">;</span>
+<span class="kn">import</span> <span class="nn">org.apache.flink.streaming.runtime.streamrecord.StreamRecord</span><span class="o">;</span>
+<span class="kn">import</span> <span class="nn">org.apache.flink.streaming.util.KeyedOneInputStreamOperatorTestHarness</span><span class="o">;</span>
+<span class="kn">import</span> <span class="nn">org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness</span><span class="o">;</span>
+
+<span class="nd">@Test</span>
+<span class="kd">public</span> <span class="kt">void</span> <span class="nf">testFlatMap</span><span class="o">()</span> <span class="kd">throws</span> <span class="n">Exception</span><span class="o">{</span>
+ <span class="n">StatefulFlatMap</span> <span class="n">statefulFlatMap</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">StatefulFlatMap</span><span class="o">();</span>
+
+ <span class="c1">// OneInputStreamOperatorTestHarness takes the input and output types as type parameters </span>
+ <span class="n">OneInputStreamOperatorTestHarness</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">String</span><span class="o">&gt;</span> <span class="n">testHarness</span> <span class="o">=</span>
+ <span class="c1">// KeyedOneInputStreamOperatorTestHarness takes three arguments:</span>
+ <span class="c1">// Flink operator object, key selector and key type</span>
+ <span class="k">new</span> <span class="n">KeyedOneInputStreamOperatorTestHarness</span><span class="o">&lt;&gt;(</span>
+ <span class="k">new</span> <span class="n">StreamFlatMap</span><span class="o">&lt;&gt;(</span><span class="n">statefulFlatMap</span><span class="o">),</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="s">&quot;1&quot;</span><span class="o">,</spa [...]
+ <span class="n">testHarness</span><span class="o">.</span><span class="na">open</span><span class="o">();</span>
+
+ <span class="c1">// test first record</span>
+ <span class="n">testHarness</span><span class="o">.</span><span class="na">processElement</span><span class="o">(</span><span class="s">&quot;world&quot;</span><span class="o">,</span> <span class="mi">10</span><span class="o">);</span>
+ <span class="n">ValueState</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">previousInput</span> <span class="o">=</span>
+ <span class="n">statefulFlatMap</span><span class="o">.</span><span class="na">getRuntimeContext</span><span class="o">().</span><span class="na">getState</span><span class="o">(</span>
+ <span class="k">new</span> <span class="n">ValueStateDescriptor</span><span class="o">&lt;&gt;(</span><span class="s">&quot;previousInput&quot;</span><span class="o">,</span> <span class="n">Types</span><span class="o">.</span><span class="na">STRING</span><span class="o">));&l [...]
+ <span class="n">String</span> <span class="n">stateValue</span> <span class="o">=</span> <span class="n">previousInput</span><span class="o">.</span><span class="na">value</span><span class="o">();</span>
+ <span class="n">Assert</span><span class="o">.</span><span class="na">assertEquals</span><span class="o">(</span>
+ <span class="n">Lists</span><span class="o">.</span><span class="na">newArrayList</span><span class="o">(</span><span class="k">new</span> <span class="n">StreamRecord</span><span class="o">&lt;&gt;(</span><span class="s">&quot;hello world&quot;</span><span class="o">,</span&g [...]
+ <span class="n">testHarness</span><span class="o">.</span><span class="na">extractOutputStreamRecords</span><span class="o">());</span>
+ <span class="n">Assert</span><span class="o">.</span><span class="na">assertEquals</span><span class="o">(</span><span class="s">&quot;world&quot;</span><span class="o">,</span> <span class="n">stateValue</span><span class="o">);</span>
+
+ <span class="c1">// test second record</span>
+ <span class="n">testHarness</span><span class="o">.</span><span class="na">processElement</span><span class="o">(</span><span class="s">&quot;parallel&quot;</span><span class="o">,</span> <span class="mi">20</span><span class="o">);</span>
+ <span class="n">Assert</span><span class="o">.</span><span class="na">assertEquals</span><span class="o">(</span>
+ <span class="n">Lists</span><span class="o">.</span><span class="na">newArrayList</span><span class="o">(</span>
+ <span class="k">new</span> <span class="n">StreamRecord</span><span class="o">&lt;&gt;(</span><span class="s">&quot;hello world&quot;</span><span class="o">,</span> <span class="mi">10</span><span class="o">),</span>
+ <span class="k">new</span> <span class="n">StreamRecord</span><span class="o">&lt;&gt;(</span><span class="s">&quot;hello parallel world&quot;</span><span class="o">,</span> <span class="mi">20</span><span class="o">)),</span>
+ <span class="n">testHarness</span><span class="o">.</span><span class="na">extractOutputStreamRecords</span><span class="o">());</span>
+ <span class="n">Assert</span><span class="o">.</span><span class="na">assertEquals</span><span class="o">(</span><span class="s">&quot;parallel&quot;</span><span class="o">,</span> <span class="n">previousInput</span><span class="o">.</span><span class="na">value</span><span class [...]
+<span class="o">}</span></code></pre></div>
+
+<p>The test harness provides many helper methods, three of which are being used here:</p>
+
+<ol>
+ <li><code>open</code>: calls the open of the <code>FlatMap</code> function with relevant parameters. It also initializes the context.</li>
+ <li><code>processElement</code>: allows users to pass an input element as well as the timestamp associated with the element.</li>
+ <li><code>extractOutputStreamRecords</code>: gets the output records along with their timestamps from the <code>Collector</code>.</li>
+</ol>
+
+<p>The test harness simplifies the unit testing for the stateful functions to a large extent.</p>
+
+<p>You might also need to check whether the state value is being set correctly. You can get the state value directly from the operator using a mechanism similar to the one used while creating the state. This is also demonstrated in the previous example.</p>
+
+<h1 id="timed-process-operators">Timed Process Operators</h1>
+
+<p>Writing tests for process functions, that work with time, is quite similar to writing tests for stateful functions because you can also use test harness.
+However, you need to take care of another aspect, which is providing timestamps for events and controlling the current time of the application. By setting the current (processing or event) time, you can trigger registered timers, which will call the <code>onTimer</code> method of the function</p>
+
+<div class="highlight"><pre><code class="language-java"><span class="kd">public</span> <span class="kd">class</span> <span class="nc">MyProcessFunction</span> <span class="kd">extends</span> <span class="n">KeyedProcessFunction</span><span class="o">&lt;</span><span class="n">String</span><sp [...]
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="nf">processElement</span><span class="o">(</span><span class="n">String</span> <span class="n">in</span><span class="o">,</span> <span class="n">Context</span> <span class="n">context</span><span class="o"& [...]
+ <span class="n">context</span><span class="o">.</span><span class="na">timerService</span><span class="o">().</span><span class="na">registerProcessingTimeTimer</span><span class="o">(</span><span class="mi">50</span><span class="o">);</span>
+ <span class="n">String</span> <span class="n">out</span> <span class="o">=</span> <span class="s">&quot;hello &quot;</span> <span class="o">+</span> <span class="n">in</span><span class="o">;</span>
+ <span class="n">collector</span><span class="o">.</span><span class="na">collect</span><span class="o">(</span><span class="n">out</span><span class="o">);</span>
+ <span class="o">}</span>
+
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="nf">onTimer</span><span class="o">(</span><span class="kt">long</span> <span class="n">timestamp</span><span class="o">,</span> <span class="n">OnTimerContext</span> <span class="n">ctx</span><span class="o" [...]
+ <span class="n">out</span><span class="o">.</span><span class="na">collect</span><span class="o">(</span><span class="n">String</span><span class="o">.</span><span class="na">format</span><span class="o">(</span><span class="s">&quot;Timer triggered at timestamp %d&quot;</span>< [...]
+ <span class="o">}</span>
+<span class="o">}</span></code></pre></div>
+
+<p>We need to test both the methods in the <code>KeyedProcessFunction</code>, i.e., <code>processElement</code> as well as <code>onTimer</code>. Using a test harness, we can control the current time of the function. Thus, we can trigger the timer at will rather than waiting for a specific time.</p>
+
+<p>Let’s take a look at the test case</p>
+
+<div class="highlight"><pre><code class="language-java"><span class="nd">@Test</span>
+<span class="kd">public</span> <span class="kt">void</span> <span class="nf">testProcessElement</span><span class="o">()</span> <span class="kd">throws</span> <span class="n">Exception</span><span class="o">{</span>
+ <span class="n">MyProcessFunction</span> <span class="n">myProcessFunction</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">MyProcessFunction</span><span class="o">();</span>
+ <span class="n">OneInputStreamOperatorTestHarness</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">String</span><span class="o">&gt;</span> <span class="n">testHarness</span> <span class="o">=</span>
+ <span class="k">new</span> <span class="n">KeyedOneInputStreamOperatorTestHarness</span><span class="o">&lt;&gt;(</span>
+ <span class="k">new</span> <span class="n">KeyedProcessOperator</span><span class="o">&lt;&gt;(</span><span class="n">myProcessFunction</span><span class="o">),</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="s">&quot;1&quot;</span><span class="o"> [...]
+
+ <span class="c1">// Function time is initialized to 0</span>
+ <span class="n">testHarness</span><span class="o">.</span><span class="na">open</span><span class="o">();</span>
+ <span class="n">testHarness</span><span class="o">.</span><span class="na">processElement</span><span class="o">(</span><span class="s">&quot;world&quot;</span><span class="o">,</span> <span class="mi">10</span><span class="o">);</span>
+
+ <span class="n">Assert</span><span class="o">.</span><span class="na">assertEquals</span><span class="o">(</span>
+ <span class="n">Lists</span><span class="o">.</span><span class="na">newArrayList</span><span class="o">(</span><span class="k">new</span> <span class="n">StreamRecord</span><span class="o">&lt;&gt;(</span><span class="s">&quot;hello world&quot;</span><span class="o">,</span&g [...]
+ <span class="n">testHarness</span><span class="o">.</span><span class="na">extractOutputStreamRecords</span><span class="o">());</span>
+<span class="o">}</span>
+
+<span class="nd">@Test</span>
+<span class="kd">public</span> <span class="kt">void</span> <span class="nf">testOnTimer</span><span class="o">()</span> <span class="kd">throws</span> <span class="n">Exception</span> <span class="o">{</span>
+ <span class="n">MyProcessFunction</span> <span class="n">myProcessFunction</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">MyProcessFunction</span><span class="o">();</span>
+ <span class="n">OneInputStreamOperatorTestHarness</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">String</span><span class="o">&gt;</span> <span class="n">testHarness</span> <span class="o">=</span>
+ <span class="k">new</span> <span class="n">KeyedOneInputStreamOperatorTestHarness</span><span class="o">&lt;&gt;(</span>
+ <span class="k">new</span> <span class="n">KeyedProcessOperator</span><span class="o">&lt;&gt;(</span><span class="n">myProcessFunction</span><span class="o">),</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="s">&quot;1&quot;</span><span class="o"> [...]
+
+ <span class="n">testHarness</span><span class="o">.</span><span class="na">open</span><span class="o">();</span>
+ <span class="n">testHarness</span><span class="o">.</span><span class="na">processElement</span><span class="o">(</span><span class="s">&quot;world&quot;</span><span class="o">,</span> <span class="mi">10</span><span class="o">);</span>
+ <span class="n">Assert</span><span class="o">.</span><span class="na">assertEquals</span><span class="o">(</span><span class="mi">1</span><span class="o">,</span> <span class="n">testHarness</span><span class="o">.</span><span class="na">numProcessingTimeTimers</span><span class="o&q [...]
- // Function time is set to 50
- testHarness.setProcessingTime(50);
- Assert.assertEquals(
- Lists.newArrayList(
- new StreamRecord<>("hello world", 10),
- new StreamRecord<>("Timer triggered at timestamp 50")),
- testHarness.extractOutputStreamRecords());
-}
-```
+ <span class="c1">// Function time is set to 50</span>
+ <span class="n">testHarness</span><span class="o">.</span><span class="na">setProcessingTime</span><span class="o">(</span><span class="mi">50</span><span class="o">);</span>
+ <span class="n">Assert</span><span class="o">.</span><span class="na">assertEquals</span><span class="o">(</span>
+ <span class="n">Lists</span><span class="o">.</span><span class="na">newArrayList</span><span class="o">(</span>
+ <span class="k">new</span> <span class="n">StreamRecord</span><span class="o">&lt;&gt;(</span><span class="s">&quot;hello world&quot;</span><span class="o">,</span> <span class="mi">10</span><span class="o">),</span>
+ <span class="k">new</span> <span class="n">StreamRecord</span><span class="o">&lt;&gt;(</span><span class="s">&quot;Timer triggered at timestamp 50&quot;</span><span class="o">)),</span>
+ <span class="n">testHarness</span><span class="o">.</span><span class="na">extractOutputStreamRecords</span><span class="o">());</span>
+<span class="o">}</span></code></pre></div>
-The mechanism to test the multi-input stream operators such as CoProcess functions is similar to the ones described in this article. You should use the TwoInput variant of the harness for these operators, such as `TwoInputStreamOperatorTestHarness`.
+<p>The mechanism to test the multi-input stream operators such as CoProcess functions is similar to the ones described in this article. You should use the TwoInput variant of the harness for these operators, such as <code>TwoInputStreamOperatorTestHarness</code>.</p>
-# Summary
+<h1 id="summary">Summary</h1>
-In the previous sections we showcased how unit testing in Apache Flink works for stateless, stateful and times-aware-operators. We hope you found the steps easy to follow and execute while developing your Flink applications. If you have any questions or feedback you can reach out to me [here](https://www.kharekartik.dev/about/) or contact the community on the [Apache Flink user mailing list](https://flink.apache.org/community.html).
+<p>In the previous sections we showcased how unit testing in Apache Flink works for stateless, stateful and times-aware-operators. We hope you found the steps easy to follow and execute while developing your Flink applications. If you have any questions or feedback you can reach out to me <a href="https://www.kharekartik.dev/about/">here</a> or contact the community on the <a href="https://flink.apache.org/community.html">Apache Flink user mail [...]
</description>
-<pubDate>Fri, 07 Feb 2020 12:00:00 +0000</pubDate>
+<pubDate>Fri, 07 Feb 2020 13:00:00 +0100</pubDate>
<link>https://flink.apache.org/news/2020/02/07/a-guide-for-unit-testing-in-apache-flink.html</link>
<guid isPermaLink="true">/news/2020/02/07/a-guide-for-unit-testing-in-apache-flink.html</guid>
</item>
<item>
<title>Apache Flink 1.9.2 Released</title>
-<description>The Apache Flink community released the second bugfix version of the Apache Flink 1.9 series.
+<description><p>The Apache Flink community released the second bugfix version of the Apache Flink 1.9 series.</p>
-This release includes 117 fixes and minor improvements for Flink 1.9.1. The list below includes a detailed list of all fixes and improvements.
+<p>This release includes 117 fixes and minor improvements for Flink 1.9.1. The list below includes a detailed list of all fixes and improvements.</p>
-We highly recommend all users to upgrade to Flink 1.9.2.
+<p>We highly recommend all users to upgrade to Flink 1.9.2.</p>
-Updated Maven dependencies:
+<p>Updated Maven dependencies:</p>
-```xml
-<dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-java</artifactId>
- <version>1.9.2</version>
-</dependency>
-<dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-streaming-java_2.11</artifactId>
- <version>1.9.2</version>
-</dependency>
-<dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-clients_2.11</artifactId>
- <version>1.9.2</version>
-</dependency>
-```
+<div class="highlight"><pre><code class="language-xml"><span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-java<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.9.2<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span>
+<span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-streaming-java_2.11<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.9.2<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span>
+<span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-clients_2.11<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.9.2<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span></code></pre></div>
-You can find the binaries on the updated [Downloads page]({{ site.baseurl }}/downloads.html).
+<p>You can find the binaries on the updated <a href="/downloads.html">Downloads page</a>.</p>
-List of resolved issues:
+<p>List of resolved issues:</p>
<h2> Sub-task
</h2>
<ul>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-12122'>FLINK-12122</a>] - Spread out tasks evenly across all available registered TaskManagers
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12122">FLINK-12122</a>] - Spread out tasks evenly across all available registered TaskManagers
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13360'>FLINK-13360</a>] - Add documentation for HBase connector for Table API &amp; SQL
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13360">FLINK-13360</a>] - Add documentation for HBase connector for Table API &amp; SQL
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13361'>FLINK-13361</a>] - Add documentation for JDBC connector for Table API &amp; SQL
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13361">FLINK-13361</a>] - Add documentation for JDBC connector for Table API &amp; SQL
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13723'>FLINK-13723</a>] - Use liquid-c for faster doc generation
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13723">FLINK-13723</a>] - Use liquid-c for faster doc generation
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13724'>FLINK-13724</a>] - Remove unnecessary whitespace from the docs&#39; sidenav
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13724">FLINK-13724</a>] - Remove unnecessary whitespace from the docs&#39; sidenav
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13725'>FLINK-13725</a>] - Use sassc for faster doc generation
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13725">FLINK-13725</a>] - Use sassc for faster doc generation
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13726'>FLINK-13726</a>] - Build docs with jekyll 4.0.0.pre.beta1
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13726">FLINK-13726</a>] - Build docs with jekyll 4.0.0.pre.beta1
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13791'>FLINK-13791</a>] - Speed up sidenav by using group_by
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13791">FLINK-13791</a>] - Speed up sidenav by using group_by
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13817'>FLINK-13817</a>] - Expose whether web submissions are enabled
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13817">FLINK-13817</a>] - Expose whether web submissions are enabled
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13818'>FLINK-13818</a>] - Check whether web submission are enabled
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13818">FLINK-13818</a>] - Check whether web submission are enabled
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14535'>FLINK-14535</a>] - Cast exception is thrown when count distinct on decimal fields
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14535">FLINK-14535</a>] - Cast exception is thrown when count distinct on decimal fields
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14735'>FLINK-14735</a>] - Improve batch schedule check input consumable performance
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14735">FLINK-14735</a>] - Improve batch schedule check input consumable performance
</li>
</ul>
-
+
<h2> Bug
</h2>
<ul>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10377'>FLINK-10377</a>] - Remove precondition in TwoPhaseCommitSinkFunction.notifyCheckpointComplete
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10377">FLINK-10377</a>] - Remove precondition in TwoPhaseCommitSinkFunction.notifyCheckpointComplete
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10435'>FLINK-10435</a>] - Client sporadically hangs after Ctrl + C
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10435">FLINK-10435</a>] - Client sporadically hangs after Ctrl + C
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-11120'>FLINK-11120</a>] - TIMESTAMPADD function handles TIME incorrectly
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11120">FLINK-11120</a>] - TIMESTAMPADD function handles TIME incorrectly
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-11835'>FLINK-11835</a>] - ZooKeeperLeaderElectionITCase.testJobExecutionOnClusterWithLeaderChange failed
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11835">FLINK-11835</a>] - ZooKeeperLeaderElectionITCase.testJobExecutionOnClusterWithLeaderChange failed
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-12342'>FLINK-12342</a>] - Yarn Resource Manager Acquires Too Many Containers
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12342">FLINK-12342</a>] - Yarn Resource Manager Acquires Too Many Containers
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-12399'>FLINK-12399</a>] - FilterableTableSource does not use filters on job run
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12399">FLINK-12399</a>] - FilterableTableSource does not use filters on job run
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13184'>FLINK-13184</a>] - Starting a TaskExecutor blocks the YarnResourceManager&#39;s main thread
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13184">FLINK-13184</a>] - Starting a TaskExecutor blocks the YarnResourceManager&#39;s main thread
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13589'>FLINK-13589</a>] - DelimitedInputFormat index error on multi-byte delimiters with whole file input splits
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13589">FLINK-13589</a>] - DelimitedInputFormat index error on multi-byte delimiters with whole file input splits
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13702'>FLINK-13702</a>] - BaseMapSerializerTest.testDuplicate fails on Travis
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13702">FLINK-13702</a>] - BaseMapSerializerTest.testDuplicate fails on Travis
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13708'>FLINK-13708</a>] - Transformations should be cleared because a table environment could execute multiple job
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13708">FLINK-13708</a>] - Transformations should be cleared because a table environment could execute multiple job
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13740'>FLINK-13740</a>] - TableAggregateITCase.testNonkeyedFlatAggregate failed on Travis
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13740">FLINK-13740</a>] - TableAggregateITCase.testNonkeyedFlatAggregate failed on Travis
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13749'>FLINK-13749</a>] - Make Flink client respect classloading policy
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13749">FLINK-13749</a>] - Make Flink client respect classloading policy
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13758'>FLINK-13758</a>] - Failed to submit JobGraph when registered hdfs file in DistributedCache
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13758">FLINK-13758</a>] - Failed to submit JobGraph when registered hdfs file in DistributedCache
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13799'>FLINK-13799</a>] - Web Job Submit Page displays stream of error message when web submit is disables in the config
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13799">FLINK-13799</a>] - Web Job Submit Page displays stream of error message when web submit is disables in the config
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13827'>FLINK-13827</a>] - Shell variable should be escaped in start-scala-shell.sh
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13827">FLINK-13827</a>] - Shell variable should be escaped in start-scala-shell.sh
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13862'>FLINK-13862</a>] - Update Execution Plan docs
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13862">FLINK-13862</a>] - Update Execution Plan docs
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13945'>FLINK-13945</a>] - Instructions for building flink-shaded against vendor repository don&#39;t work
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13945">FLINK-13945</a>] - Instructions for building flink-shaded against vendor repository don&#39;t work
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13969'>FLINK-13969</a>] - Resuming Externalized Checkpoint (rocks, incremental, scale down) end-to-end test fails on Travis
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13969">FLINK-13969</a>] - Resuming Externalized Checkpoint (rocks, incremental, scale down) end-to-end test fails on Travis
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13995'>FLINK-13995</a>] - Fix shading of the licence information of netty
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13995">FLINK-13995</a>] - Fix shading of the licence information of netty
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13999'>FLINK-13999</a>] - Correct the documentation of MATCH_RECOGNIZE
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13999">FLINK-13999</a>] - Correct the documentation of MATCH_RECOGNIZE
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14066'>FLINK-14066</a>] - Pyflink building failure in master and 1.9.0 version
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14066">FLINK-14066</a>] - Pyflink building failure in master and 1.9.0 version
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14074'>FLINK-14074</a>] - MesosResourceManager can&#39;t create new taskmanagers in Session Cluster Mode.
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14074">FLINK-14074</a>] - MesosResourceManager can&#39;t create new taskmanagers in Session Cluster Mode.
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14175'>FLINK-14175</a>] - Upgrade KPL version in flink-connector-kinesis to fix application OOM
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14175">FLINK-14175</a>] - Upgrade KPL version in flink-connector-kinesis to fix application OOM
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14200'>FLINK-14200</a>] - Temporal Table Function Joins do not work on Tables (only TableSources) on the query side
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14200">FLINK-14200</a>] - Temporal Table Function Joins do not work on Tables (only TableSources) on the query side
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14235'>FLINK-14235</a>] - Kafka010ProducerITCase&gt;KafkaProducerTestBase.testOneToOneAtLeastOnceCustomOperator fails on travis
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14235">FLINK-14235</a>] - Kafka010ProducerITCase&gt;KafkaProducerTestBase.testOneToOneAtLeastOnceCustomOperator fails on travis
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14315'>FLINK-14315</a>] - NPE with JobMaster.disconnectTaskManager
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14315">FLINK-14315</a>] - NPE with JobMaster.disconnectTaskManager
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14337'>FLINK-14337</a>] - HistoryServer does not handle NPE on corruped archives properly
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14337">FLINK-14337</a>] - HistoryServer does not handle NPE on corruped archives properly
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14347'>FLINK-14347</a>] - YARNSessionFIFOITCase.checkForProhibitedLogContents found a log with prohibited string
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14347">FLINK-14347</a>] - YARNSessionFIFOITCase.checkForProhibitedLogContents found a log with prohibited string
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14355'>FLINK-14355</a>] - Example code in state processor API docs doesn&#39;t compile
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14355">FLINK-14355</a>] - Example code in state processor API docs doesn&#39;t compile
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14370'>FLINK-14370</a>] - KafkaProducerAtLeastOnceITCase&gt;KafkaProducerTestBase.testOneToOneAtLeastOnceRegularSink fails on Travis
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14370">FLINK-14370</a>] - KafkaProducerAtLeastOnceITCase&gt;KafkaProducerTestBase.testOneToOneAtLeastOnceRegularSink fails on Travis
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14382'>FLINK-14382</a>] - Incorrect handling of FLINK_PLUGINS_DIR on Yarn
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14382">FLINK-14382</a>] - Incorrect handling of FLINK_PLUGINS_DIR on Yarn
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14398'>FLINK-14398</a>] - Further split input unboxing code into separate methods
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14398">FLINK-14398</a>] - Further split input unboxing code into separate methods
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14413'>FLINK-14413</a>] - Shade-plugin ApacheNoticeResourceTransformer uses platform-dependent encoding
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14413">FLINK-14413</a>] - Shade-plugin ApacheNoticeResourceTransformer uses platform-dependent encoding
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14434'>FLINK-14434</a>] - Dispatcher#createJobManagerRunner should not start JobManagerRunner
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14434">FLINK-14434</a>] - Dispatcher#createJobManagerRunner should not start JobManagerRunner
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14445'>FLINK-14445</a>] - Python module build failed when making sdist
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14445">FLINK-14445</a>] - Python module build failed when making sdist
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14447'>FLINK-14447</a>] - Network metrics doc table render confusion
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14447">FLINK-14447</a>] - Network metrics doc table render confusion
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14459'>FLINK-14459</a>] - Python module build hangs
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14459">FLINK-14459</a>] - Python module build hangs
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14524'>FLINK-14524</a>] - PostgreSQL JDBC sink generates invalid SQL in upsert mode
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14524">FLINK-14524</a>] - PostgreSQL JDBC sink generates invalid SQL in upsert mode
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14547'>FLINK-14547</a>] - UDF cannot be in the join condition in blink planner
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14547">FLINK-14547</a>] - UDF cannot be in the join condition in blink planner
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14561'>FLINK-14561</a>] - Don&#39;t write FLINK_PLUGINS_DIR ENV variable to Flink configuration
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14561">FLINK-14561</a>] - Don&#39;t write FLINK_PLUGINS_DIR ENV variable to Flink configuration
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14562'>FLINK-14562</a>] - RMQSource leaves idle consumer after closing
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14562">FLINK-14562</a>] - RMQSource leaves idle consumer after closing
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14574'>FLINK-14574</a>] - flink-s3-fs-hadoop doesn&#39;t work with plugins mechanism
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14574">FLINK-14574</a>] - flink-s3-fs-hadoop doesn&#39;t work with plugins mechanism
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14589'>FLINK-14589</a>] - Redundant slot requests with the same AllocationID leads to inconsistent slot table
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14589">FLINK-14589</a>] - Redundant slot requests with the same AllocationID leads to inconsistent slot table
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14641'>FLINK-14641</a>] - Fix description of metric `fullRestarts`
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14641">FLINK-14641</a>] - Fix description of metric `fullRestarts`
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14673'>FLINK-14673</a>] - Shouldn&#39;t expect HMS client to throw NoSuchObjectException for non-existing function
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14673">FLINK-14673</a>] - Shouldn&#39;t expect HMS client to throw NoSuchObjectException for non-existing function
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14683'>FLINK-14683</a>] - RemoteStreamEnvironment&#39;s construction function has a wrong method
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14683">FLINK-14683</a>] - RemoteStreamEnvironment&#39;s construction function has a wrong method
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14701'>FLINK-14701</a>] - Slot leaks if SharedSlotOversubscribedException happens
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14701">FLINK-14701</a>] - Slot leaks if SharedSlotOversubscribedException happens
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14784'>FLINK-14784</a>] - CsvTableSink miss delimiter when row start with null member
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14784">FLINK-14784</a>] - CsvTableSink miss delimiter when row start with null member
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14817'>FLINK-14817</a>] - &quot;Streaming Aggregation&quot; document contains misleading code examples
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14817">FLINK-14817</a>] - &quot;Streaming Aggregation&quot; document contains misleading code examples
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14846'>FLINK-14846</a>] - Correct the default writerbuffer size documentation of RocksDB
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14846">FLINK-14846</a>] - Correct the default writerbuffer size documentation of RocksDB
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14910'>FLINK-14910</a>] - DisableAutoGeneratedUIDs fails on keyBy
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14910">FLINK-14910</a>] - DisableAutoGeneratedUIDs fails on keyBy
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14930'>FLINK-14930</a>] - OSS Filesystem Uses Wrong Shading Prefix
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14930">FLINK-14930</a>] - OSS Filesystem Uses Wrong Shading Prefix
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14949'>FLINK-14949</a>] - Task cancellation can be stuck against out-of-thread error
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14949">FLINK-14949</a>] - Task cancellation can be stuck against out-of-thread error
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14951'>FLINK-14951</a>] - State TTL backend end-to-end test fail when taskManager has multiple slot
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14951">FLINK-14951</a>] - State TTL backend end-to-end test fail when taskManager has multiple slot
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14953'>FLINK-14953</a>] - Parquet table source should use schema type to build FilterPredicate
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14953">FLINK-14953</a>] - Parquet table source should use schema type to build FilterPredicate
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14960'>FLINK-14960</a>] - Dependency shading of table modules test fails on Travis
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14960">FLINK-14960</a>] - Dependency shading of table modules test fails on Travis
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14976'>FLINK-14976</a>] - Cassandra Connector leaks Semaphore on Throwable; hangs on close
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14976">FLINK-14976</a>] - Cassandra Connector leaks Semaphore on Throwable; hangs on close
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-15001'>FLINK-15001</a>] - The digest of sub-plan reuse should contain retraction traits for stream physical nodes
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15001">FLINK-15001</a>] - The digest of sub-plan reuse should contain retraction traits for stream physical nodes
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-15013'>FLINK-15013</a>] - Flink (on YARN) sometimes needs too many slots
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15013">FLINK-15013</a>] - Flink (on YARN) sometimes needs too many slots
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-15030'>FLINK-15030</a>] - Potential deadlock for bounded blocking ResultPartition.
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15030">FLINK-15030</a>] - Potential deadlock for bounded blocking ResultPartition.
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-15036'>FLINK-15036</a>] - Container startup error will be handled out side of the YarnResourceManager&#39;s main thread
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15036">FLINK-15036</a>] - Container startup error will be handled out side of the YarnResourceManager&#39;s main thread
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-15063'>FLINK-15063</a>] - Input group and output group of the task metric are reversed
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15063">FLINK-15063</a>] - Input group and output group of the task metric are reversed
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-15065'>FLINK-15065</a>] - RocksDB configurable options doc description error
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15065">FLINK-15065</a>] - RocksDB configurable options doc description error
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-15076'>FLINK-15076</a>] - Source thread should be interrupted during the Task cancellation
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15076">FLINK-15076</a>] - Source thread should be interrupted during the Task cancellation
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-15234'>FLINK-15234</a>] - Hive table created from flink catalog table shouldn&#39;t have null properties in parameters
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15234">FLINK-15234</a>] - Hive table created from flink catalog table shouldn&#39;t have null properties in parameters
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-15240'>FLINK-15240</a>] - is_generic key is missing for Flink table stored in HiveCatalog
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15240">FLINK-15240</a>] - is_generic key is missing for Flink table stored in HiveCatalog
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-15259'>FLINK-15259</a>] - HiveInspector.toInspectors() should convert Flink constant to Hive constant
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15259">FLINK-15259</a>] - HiveInspector.toInspectors() should convert Flink constant to Hive constant
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-15266'>FLINK-15266</a>] - NPE in blink planner code gen
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15266">FLINK-15266</a>] - NPE in blink planner code gen
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-15361'>FLINK-15361</a>] - ParquetTableSource should pass predicate in projectFields
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15361">FLINK-15361</a>] - ParquetTableSource should pass predicate in projectFields
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-15412'>FLINK-15412</a>] - LocalExecutorITCase#testParameterizedTypes failed in travis
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15412">FLINK-15412</a>] - LocalExecutorITCase#testParameterizedTypes failed in travis
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-15413'>FLINK-15413</a>] - ScalarOperatorsTest failed in travis
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15413">FLINK-15413</a>] - ScalarOperatorsTest failed in travis
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-15418'>FLINK-15418</a>] - StreamExecMatchRule not set FlinkRelDistribution
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15418">FLINK-15418</a>] - StreamExecMatchRule not set FlinkRelDistribution
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-15421'>FLINK-15421</a>] - GroupAggsHandler throws java.time.LocalDateTime cannot be cast to java.sql.Timestamp
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15421">FLINK-15421</a>] - GroupAggsHandler throws java.time.LocalDateTime cannot be cast to java.sql.Timestamp
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-15435'>FLINK-15435</a>] - ExecutionConfigTests.test_equals_and_hash in pyFlink fails when cpu core numbers is 6
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15435">FLINK-15435</a>] - ExecutionConfigTests.test_equals_and_hash in pyFlink fails when cpu core numbers is 6
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-15443'>FLINK-15443</a>] - Use JDBC connector write FLOAT value occur ClassCastException
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15443">FLINK-15443</a>] - Use JDBC connector write FLOAT value occur ClassCastException
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-15478'>FLINK-15478</a>] - FROM_BASE64 code gen type wrong
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15478">FLINK-15478</a>] - FROM_BASE64 code gen type wrong
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-15489'>FLINK-15489</a>] - WebUI log refresh not working
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15489">FLINK-15489</a>] - WebUI log refresh not working
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-15522'>FLINK-15522</a>] - Misleading root cause exception when cancelling the job
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15522">FLINK-15522</a>] - Misleading root cause exception when cancelling the job
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-15523'>FLINK-15523</a>] - ConfigConstants generally excluded from japicmp
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15523">FLINK-15523</a>] - ConfigConstants generally excluded from japicmp
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-15543'>FLINK-15543</a>] - Apache Camel not bundled but listed in flink-dist NOTICE
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15543">FLINK-15543</a>] - Apache Camel not bundled but listed in flink-dist NOTICE
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-15549'>FLINK-15549</a>] - Integer overflow in SpillingResettableMutableObjectIterator
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15549">FLINK-15549</a>] - Integer overflow in SpillingResettableMutableObjectIterator
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-15577'>FLINK-15577</a>] - WindowAggregate RelNodes missing Window specs in digest
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15577">FLINK-15577</a>] - WindowAggregate RelNodes missing Window specs in digest
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-15615'>FLINK-15615</a>] - Docs: wrong guarantees stated for the file sink
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15615">FLINK-15615</a>] - Docs: wrong guarantees stated for the file sink
</li>
</ul>
-
+
<h2> Improvement
</h2>
<ul>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-11135'>FLINK-11135</a>] - Reorder Hadoop config loading in HadoopUtils
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11135">FLINK-11135</a>] - Reorder Hadoop config loading in HadoopUtils
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-12848'>FLINK-12848</a>] - Method equals() in RowTypeInfo should consider fieldsNames
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12848">FLINK-12848</a>] - Method equals() in RowTypeInfo should consider fieldsNames
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13729'>FLINK-13729</a>] - Update website generation dependencies
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13729">FLINK-13729</a>] - Update website generation dependencies
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14008'>FLINK-14008</a>] - Auto-generate binary licensing
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14008">FLINK-14008</a>] - Auto-generate binary licensing
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14104'>FLINK-14104</a>] - Bump Jackson to 2.10.1
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14104">FLINK-14104</a>] - Bump Jackson to 2.10.1
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14123'>FLINK-14123</a>] - Lower the default value of taskmanager.memory.fraction
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14123">FLINK-14123</a>] - Lower the default value of taskmanager.memory.fraction
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14206'>FLINK-14206</a>] - Let fullRestart metric count fine grained restarts as well
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14206">FLINK-14206</a>] - Let fullRestart metric count fine grained restarts as well
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14215'>FLINK-14215</a>] - Add Docs for TM and JM Environment Variable Setting
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14215">FLINK-14215</a>] - Add Docs for TM and JM Environment Variable Setting
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14251'>FLINK-14251</a>] - Add FutureUtils#forward utility
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14251">FLINK-14251</a>] - Add FutureUtils#forward utility
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14334'>FLINK-14334</a>] - ElasticSearch docs refer to non-existent ExceptionUtils.containsThrowable
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14334">FLINK-14334</a>] - ElasticSearch docs refer to non-existent ExceptionUtils.containsThrowable
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14335'>FLINK-14335</a>] - ExampleIntegrationTest in testing docs is incorrect
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14335">FLINK-14335</a>] - ExampleIntegrationTest in testing docs is incorrect
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14408'>FLINK-14408</a>] - In OldPlanner, UDF open method can not be invoke when SQL is optimized
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14408">FLINK-14408</a>] - In OldPlanner, UDF open method can not be invoke when SQL is optimized
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14557'>FLINK-14557</a>] - Clean up the package of py4j
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14557">FLINK-14557</a>] - Clean up the package of py4j
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14639'>FLINK-14639</a>] - Metrics User Scope docs refer to wrong class
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14639">FLINK-14639</a>] - Metrics User Scope docs refer to wrong class
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14646'>FLINK-14646</a>] - Check non-null for key in KeyGroupStreamPartitioner
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14646">FLINK-14646</a>] - Check non-null for key in KeyGroupStreamPartitioner
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14825'>FLINK-14825</a>] - Rework state processor api documentation
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14825">FLINK-14825</a>] - Rework state processor api documentation
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14995'>FLINK-14995</a>] - Kinesis NOTICE is incorrect
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14995">FLINK-14995</a>] - Kinesis NOTICE is incorrect
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-15113'>FLINK-15113</a>] - fs.azure.account.key not hidden from global configuration
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15113">FLINK-15113</a>] - fs.azure.account.key not hidden from global configuration
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-15554'>FLINK-15554</a>] - Bump jetty-util-ajax to 9.3.24
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15554">FLINK-15554</a>] - Bump jetty-util-ajax to 9.3.24
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-15657'>FLINK-15657</a>] - Fix the python table api doc link in Python API tutorial
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15657">FLINK-15657</a>] - Fix the python table api doc link in Python API tutorial
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-15700'>FLINK-15700</a>] - Improve Python API Tutorial doc
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15700">FLINK-15700</a>] - Improve Python API Tutorial doc
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-15726'>FLINK-15726</a>] - Fixing error message in StreamExecTableSourceScan
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15726">FLINK-15726</a>] - Fixing error message in StreamExecTableSourceScan
</li>
</ul>
</description>
-<pubDate>Thu, 30 Jan 2020 12:00:00 +0000</pubDate>
+<pubDate>Thu, 30 Jan 2020 13:00:00 +0100</pubDate>
<link>https://flink.apache.org/news/2020/01/30/release-1.9.2.html</link>
<guid isPermaLink="true">/news/2020/01/30/release-1.9.2.html</guid>
</item>
<item>
<title>State Unlocked: Interacting with State in Apache Flink</title>
-<description># Introduction
+<description><h1 id="introduction">Introduction</h1>
-With stateful stream-processing becoming the norm for complex event-driven applications and real-time analytics, [Apache Flink](https://flink.apache.org/) is often the backbone for running business logic and managing an organization’s most valuable asset — its data — as application state in Flink.
+<p>With stateful stream-processing becoming the norm for complex event-driven applications and real-time analytics, <a href="https://flink.apache.org/">Apache Flink</a> is often the backbone for running business logic and managing an organization’s most valuable asset — its data — as application state in Flink.</p>
-In order to provide a state-of-the-art experience to Flink developers, the Apache Flink community makes significant efforts to provide the safety and future-proof guarantees organizations need while managing state in Flink. In particular, Flink developers should have sufficient means to access and modify their state, as well as making bootstrapping state with existing data from external systems a piece-of-cake. These efforts span multiple Flink major releases and consist of the following:
+<p>In order to provide a state-of-the-art experience to Flink developers, the Apache Flink community makes significant efforts to provide the safety and future-proof guarantees organizations need while managing state in Flink. In particular, Flink developers should have sufficient means to access and modify their state, as well as making bootstrapping state with existing data from external systems a piece-of-cake. These efforts span multiple Flink major releases and consist of the [...]
-1. Evolvable state schema in Apache Flink
-2. Flexibility in swapping state backends, and
-3. The State processor API, an offline tool to read, write and modify state in Flink
+<ol>
+ <li>Evolvable state schema in Apache Flink</li>
+ <li>Flexibility in swapping state backends, and</li>
+ <li>The State processor API, an offline tool to read, write and modify state in Flink</li>
+</ol>
-This post discusses the community’s efforts related to state management in Flink, provides some practical examples of how the different features and APIs can be utilized and covers some future ideas for new and improved ways of managing state in Apache Flink.
+<p>This post discusses the community’s efforts related to state management in Flink, provides some practical examples of how the different features and APIs can be utilized and covers some future ideas for new and improved ways of managing state in Apache Flink.</p>
+<h1 id="stream-processing-what-is-state">Stream processing: What is State?</h1>
-# Stream processing: What is State?
+<p>To set the tone for the remaining of the post, let us first try to explain the very definition of state in stream processing. When it comes to stateful stream processing, state comprises of the information that an application or stream processing engine will remember across events and streams as more realtime (unbounded) and/or offline (bounded) data flow through the system. Most trivial applications are inherently stateful; even the example of a simple COUNT operation, whereby [...]
-To set the tone for the remaining of the post, let us first try to explain the very definition of state in stream processing. When it comes to stateful stream processing, state comprises of the information that an application or stream processing engine will remember across events and streams as more realtime (unbounded) and/or offline (bounded) data flow through the system. Most trivial applications are inherently stateful; even the example of a simple COUNT operation, whereby when coun [...]
-
-To better understand how Flink manages state, one can think of Flink like a three-layered state abstraction, as illustrated in the diagram below.
+<p>To better understand how Flink manages state, one can think of Flink like a three-layered state abstraction, as illustrated in the diagram below.</p>
<center>
-<img src="{{ site.baseurl }}/img/blog/2020-01-29-state-unlocked-interacting-with-state-in-apache-flink/managing-state-in-flink-visual-1.png" width="600px" alt="State in Apache Flink"/>
+<img src="/img/blog/2020-01-29-state-unlocked-interacting-with-state-in-apache-flink/managing-state-in-flink-visual-1.png" width="600px" alt="State in Apache Flink" />
</center>
-<br>
+<p><br /></p>
-On the top layer, sits the Flink user code, for example, a `KeyedProcessFunction` that contains some value state. This is a simple variable whose value state annotations makes it automatically fault-tolerant, re-scalable and queryable by the runtime. These variables are backed by the configured state backend that sits either on-heap or on-disk (RocksDB State Backend) and provides data locality, proximity to the computation and speed when it comes to per-record computations. Finally, when [...]
+<p>On the top layer, sits the Flink user code, for example, a <code>KeyedProcessFunction</code> that contains some value state. This is a simple variable whose value state annotations makes it automatically fault-tolerant, re-scalable and queryable by the runtime. These variables are backed by the configured state backend that sits either on-heap or on-disk (RocksDB State Backend) and provides data locality, proximity to the computation and speed when it comes to per-re [...]
-A savepoint is a snapshot of the distributed, global state of an application at a logical point-in-time and is stored in an external distributed file system or blob storage such as HDFS, or S3. Upon upgrading an application or implementing a code change — such as adding a new operator or changing a field — the Flink job can restart by re-loading the application state from the savepoint into the state backend, making it local and available for the computation and continue processing as i [...]
+<p>A savepoint is a snapshot of the distributed, global state of an application at a logical point-in-time and is stored in an external distributed file system or blob storage such as HDFS, or S3. Upon upgrading an application or implementing a code change — such as adding a new operator or changing a field — the Flink job can restart by re-loading the application state from the savepoint into the state backend, making it local and available for the computation and continue proces [...]
<center>
-<img src="{{ site.baseurl }}/img/blog/2020-01-29-state-unlocked-interacting-with-state-in-apache-flink/managing-state-in-flink-visual-2.png" width="600px" alt="State in Apache Flink"/>
+<img src="/img/blog/2020-01-29-state-unlocked-interacting-with-state-in-apache-flink/managing-state-in-flink-visual-2.png" width="600px" alt="State in Apache Flink" />
</center>
-<br>
+<p><br /></p>
<div class="alert alert-info">
It is important to remember here that <b>state is one of the most valuable components of a Flink application</b> carrying all the information about both where you are now and where you are going. State is among the most long-lived components in a Flink service since it can be carried across jobs, operators, configurations, new features and bug fixes.
</div>
-# Schema Evolution with Apache Flink
+<h1 id="schema-evolution-with-apache-flink">Schema Evolution with Apache Flink</h1>
+
+<p>In the previous section, we explained how state is stored and persisted in a Flink application. Let’s now take a look at what happens when evolving state in a stateful Flink streaming application becomes necessary.</p>
+
+<p>Imagine an Apache Flink application that implements a <code>KeyedProcessFunction</code> and contains some <code>ValueState</code>. As illustrated below, within the state descriptor, when registering the type, Flink users specify their <code>TypeInformation</code> that informs Flink about how to serialize the bytes and represents Flink’s internal type system, used to serialize data when shipped across the network or stored in state backends. Fl [...]
+
+<h2 id="state-registration-with-built-in-serialization-in-apache-flink">State registration with built-in serialization in Apache Flink</h2>
+
+<div class="highlight"><pre><code class="language-java"><span class="kd">public</span> <span class="kd">class</span> <span class="nc">MyFunction</span> <span class="kd">extends</span> <span class="n">KeyedProcessFunction</span><span class="o">&lt;</span><span class="n">Key</span><span class=& [...]
+<span class="err"></span>
+ <span class="kd">private</span> <span class="kd">transient</span> <span class="n">ValueState</span><span class="o">&lt;</span><span class="n">MyState</span><span class="o">&gt;</span> <span class="n">valueState</span><span class="o">;</span>
+<span class="err"></span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="nf">open</span><span class="o">(</span><span class="n">Configuration</span> <span class="n">parameters</span><span class="o">)</span> <span class="o">{</span>
+ <span class="n">ValueStateDescriptor</span><span class="o">&lt;</span><span class="n">MyState</span><span class="o">&gt;</span> <span class="n">descriptor</span> <span class="o">=</span>
+ <span class="k">new</span> <span class="n">ValueStateDescriptor</span><span class="o">&lt;&gt;(</span><span class="s">&quot;my-state&quot;</span><span class="o">,</span> <span class="n">TypeInformation</span><span class="o">.</span><span class="na">of</span><span class="o">(< [...]
+<span class="err"></span>
+ <span class="n">valueState</span> <span class="o">=</span> <span class="n">getRuntimeContext</span><span class="o">().</span><span class="na">getState</span><span class="o">(</span><span class="n">descriptor</span><span class="o">);</span>
+ <span class="o">}</span>
+<span class="o">}</span></code></pre></div>
+
+<p>Typically, evolving the schema of an application’s state happens because of some business logic change (adding or dropping fields or changing data types). In all cases, the schema is determined by means of its serializer, and can be thought of in terms of an alter table statement when compared with a database. When a state variable is first introduced it is like running a <code>CREATE_TABLE</code> command, there is a lot of freedom with its execution. However, having [...]
-In the previous section, we explained how state is stored and persisted in a Flink application. Let’s now take a look at what happens when evolving state in a stateful Flink streaming application becomes necessary.
+<p><a href="https://flink.apache.org/downloads.html#apache-flink-182">Flink 1.8</a> comes with built-in support for <a href="https://avro.apache.org/">Apache Avro</a> (specifically the <a href="https://avro.apache.org/docs/1.7.7/spec.html">1.7.7 specification</a>) and evolves state schema according to Avro specifications by adding and removing types or even by swapping between generic and specific Avro record types.& [...]
-Imagine an Apache Flink application that implements a `KeyedProcessFunction` and contains some `ValueState`. As illustrated below, within the state descriptor, when registering the type, Flink users specify their `TypeInformation` that informs Flink about how to serialize the bytes and represents Flink’s internal type system, used to serialize data when shipped across the network or stored in state backends. Flink’s type system has built-in support for all the basic types such as longs, [...]
+<p>In <a href="https://flink.apache.org/downloads.html#apache-flink-191">Flink 1.9</a> the community added support for schema evolution for POJOs, including the ability to remove existing fields from POJO types or add new fields. The POJO schema evolution tends to be less flexible — when compared to Avro — since it is not possible to change neither the declared field types nor the class name of a POJO type, including its namespace.</p>
-## State registration with built-in serialization in Apache Flink
+<p>With the community’s efforts related to schema evolution, Flink developers can now expect out-of-the-box support for both Avro and POJO formats, with backwards compatibility for all Flink state backends. Future work revolves around adding support for Scala Case Classes, Tuples and other formats. Make sure to subscribe to the <a href="https://flink.apache.org/community.html">Flink mailing list</a> to contribute and stay on top of any upcoming additions in th [...]
-```java
-public class MyFunction extends KeyedProcessFunction<Key, Input, Output> {
-
- private transient ValueState<MyState> valueState;
-
- public void open(Configuration parameters) {
- ValueStateDescriptor<MyState> descriptor =
- new ValueStateDescriptor<>("my-state", TypeInformation.of(MyState.class));
-
- valueState = getRuntimeContext().getState(descriptor);
- }
-}
-```
-
-Typically, evolving the schema of an application’s state happens because of some business logic change (adding or dropping fields or changing data types). In all cases, the schema is determined by means of its serializer, and can be thought of in terms of an alter table statement when compared with a database. When a state variable is first introduced it is like running a `CREATE_TABLE` command, there is a lot of freedom with its execution. However, having data in that table (registered [...]
-
-[Flink 1.8](https://flink.apache.org/downloads.html#apache-flink-182) comes with built-in support for [Apache Avro](https://avro.apache.org/) (specifically the [1.7.7 specification](https://avro.apache.org/docs/1.7.7/spec.html)) and evolves state schema according to Avro specifications by adding and removing types or even by swapping between generic and specific Avro record types.
-
-In [Flink 1.9](https://flink.apache.org/downloads.html#apache-flink-191) the community added support for schema evolution for POJOs, including the ability to remove existing fields from POJO types or add new fields. The POJO schema evolution tends to be less flexible — when compared to Avro — since it is not possible to change neither the declared field types nor the class name of a POJO type, including its namespace.
-
-With the community’s efforts related to schema evolution, Flink developers can now expect out-of-the-box support for both Avro and POJO formats, with backwards compatibility for all Flink state backends. Future work revolves around adding support for Scala Case Classes, Tuples and other formats. Make sure to subscribe to the [Flink mailing list](https://flink.apache.org/community.html) to contribute and stay on top of any upcoming additions in this space.
-
-## Peeking Under the Hood
-
-Now that we have explained how schema evolution in Flink works, let’s describe the challenges of performing schema serialization with Flink under the hood. Flink considers state as a core part of its API stability, in a way that developers should always be able to take a savepoint from one version of Flink and restart it on the next. With schema evolution, every migration needs to be backwards compatible and also compatible with the different state backends. While in the Flink code the s [...]
-
-For instance, the heap state backend supports lazy serialization and eager deserialization, making the per-record code path always working with Java objects, serializing on a background thread. When restoring, Flink will eagerly deserialize all the data and then start the user code. If a developer plugs in a new serializer, the deserialization happens before Flink ever receives the information.
-
-The RocksDB state backend behaves in the exact opposite manner: it supports eager serialization — because of items being stored on disk and RocksDB only consuming byte arrays. RocksDB provides lazy deserialization simply by downloading files to the local disk, making Flink unaware of what the bytes mean until a serializer is registered.
-
-An additional challenge stems from the fact that different versions of user code contain different classes on their classpath making the serializer used to write into a savepoint likely potentially unavailable at runtime.
-
-To overcome the previously mentioned challenges, we introduced what we call `TypeSerializerSnapshot`. The `TypeSerializerSnapshot` stores the configuration of the writer serializer in the snapshot. When restoring it will use that configuration to read back the previous state and check its compatibility with the current version. Using such operation allows Flink to:
-
-* Read the configuration used to write out a snapshot
-* Consume the new user code
-* Check if both items above are compatible
-* Consume the bytes from the snapshot and move forward or alert the user otherwise
-
-```java
-public interface TypeSerializerSnapshot<T> {
-
- int getCurrentVersion();
-
- void writeSnapshot(DataOutputView out) throws IOException;
-
- void readSnapshot(
- int readVersion,
- DataInputView in,
- ClassLoader userCodeClassLoader) throws IOException;
-
- TypeSerializer<T> restoreSerializer();
-
- TypeSerializerSchemaCompatibility<T> resolveSchemaCompatibility(
- TypeSerializer<T> newSerializer);
-}
-```
-
-## Implementing Apache Avro Serialization in Flink
-
-Apache Avro is a data serialization format that has very well-defined schema migration semantics and supports both reader and writer schemas. During normal Flink execution the reader and writer schemas will be the same. However, when upgrading an application they may be different and with schema evolution, Flink will be able to migrate objects with their schemas.
-
-```java
-public class AvroSerializerSnapshot<T> implements TypeSerializerSnapshot<T> {
- private Schema runtimeSchema;
- private Schema previousSchema;
-
- @SuppressWarnings("WeakerAccess")
- public AvroSerializerSnapshot() { }
-
- AvroSerializerSnapshot(Schema schema) {
- this.runtimeSchema = schema;
- }
-```
-
-This is a sketch of our Avro serializer. It uses the provided schemas and delegates to Apache Avro for all (de)-serialization. Let’s take a look at one possible implementation of a `TypeSerializerSnapshot` that supports schema migration for Avro.
-
-
-# Writing out the snapshot
-
-When serializing out the snapshot, the snapshot configuration will write two pieces of information; the current snapshot configuration version and the serializer configuration.
-
-```java
- @Override
- public int getCurrentVersion() {
- return 1;
- }
-
- @Override
- public void writeSnapshot(DataOutputView out) throws IOException {
- out.writeUTF(runtimeSchema.toString(false));
- }
-```
+<h2 id="peeking-under-the-hood">Peeking Under the Hood</h2>
-The version is used to version the snapshot configuration object itself while the `writeSnapshot` method writes out all the information we need to understand the current format; the runtime schema.
+<p>Now that we have explained how schema evolution in Flink works, let’s describe the challenges of performing schema serialization with Flink under the hood. Flink considers state as a core part of its API stability, in a way that developers should always be able to take a savepoint from one version of Flink and restart it on the next. With schema evolution, every migration needs to be backwards compatible and also compatible with the different state backends. While in the Flink c [...]
-```java
- @Override
- public void readSnapshot(
- int readVersion,
- DataInputView in,
- ClassLoader userCodeClassLoader) throws IOException {
-
- assert readVersion == 1;
- final String previousSchemaDefinition = in.readUTF();
- this.previousSchema = parseAvroSchema(previousSchemaDefinition);
- this.runtimeType = findClassOrFallbackToGeneric(
- userCodeClassLoader,
- previousSchema.getFullName());
-
- this.runtimeSchema = tryExtractAvroSchema(userCodeClassLoader, runtimeType);
- }
-```
-Now when Flink restores it is able to read back in the writer schema used to serialize the data. The current runtime schema is discovered on the class path using some Java reflection magic.
+<p>For instance, the heap state backend supports lazy serialization and eager deserialization, making the per-record code path always working with Java objects, serializing on a background thread. When restoring, Flink will eagerly deserialize all the data and then start the user code. If a developer plugs in a new serializer, the deserialization happens before Flink ever receives the information.</p>
-Once we have both of these we can compare them for compatibility. Perhaps nothing has changed and the schemas are compatible as is.
+<p>The RocksDB state backend behaves in the exact opposite manner: it supports eager serialization — because of items being stored on disk and RocksDB only consuming byte arrays. RocksDB provides lazy deserialization simply by downloading files to the local disk, making Flink unaware of what the bytes mean until a serializer is registered.</p>
-```java
- @Override
- public TypeSerializerSchemaCompatibility<T> resolveSchemaCompatibility(
- TypeSerializer<T> newSerializer) {
-
- if (!(newSerializer instanceof AvroSerializer)) {
- return TypeSerializerSchemaCompatibility.incompatible();
- }
-
- if (Objects.equals(previousSchema, runtimeSchema)) {
- return TypeSerializerSchemaCompatibility.compatibleAsIs();
- }
-```
-
-Otherwise, the schemas are compared using Avro’s compatibility checks and they may either be compatible with a migration or incompatible.
-
-```java
- final SchemaPairCompatibility compatibility = SchemaCompatibility
- .checkReaderWriterCompatibility(previousSchema, runtimeSchema);
-
- return avroCompatibilityToFlinkCompatibility(compatibility);
- }
-```
-
-If they are compatible with migration then Flink will restore a new serializer that can read the old schema and deserialize into the new runtime type which is in effect a migration.
-
-```java
- @Override
- public TypeSerializer<T> restoreSerializer() {
- if (previousSchema != null) {
- return new AvroSerializer<>(runtimeType, runtimeSchema, previousSchema);
- } else {
- return new AvroSerializer<>(runtimeType, runtimeSchema, runtimeSchema);
- }
- }
-}
-```
+<p>An additional challenge stems from the fact that different versions of user code contain different classes on their classpath making the serializer used to write into a savepoint likely potentially unavailable at runtime.</p>
-# The State Processor API: Reading, writing and modifying Flink state
+<p>To overcome the previously mentioned challenges, we introduced what we call <code>TypeSerializerSnapshot</code>. The <code>TypeSerializerSnapshot</code> stores the configuration of the writer serializer in the snapshot. When restoring it will use that configuration to read back the previous state and check its compatibility with the current version. Using such operation allows Flink to:</p>
-The State Processor API allows reading from and writing to Flink savepoints. Some of the interesting use cases it can be used for are:
+<ul>
+ <li>Read the configuration used to write out a snapshot</li>
+ <li>Consume the new user code</li>
+ <li>Check if both items above are compatible</li>
+ <li>Consume the bytes from the snapshot and move forward or alert the user otherwise</li>
+</ul>
-* Analyzing state for interesting patterns
-* Troubleshooting or auditing jobs by checking for state discrepancies
-* Bootstrapping state for new applications
-* Modifying savepoints such as:
- * Changing the maximum parallelism of a savepoint after deploying a Flink job
- * Introducing breaking schema updates to a Flink application
- * Correcting invalid state in a Flink savepoint
+<div class="highlight"><pre><code class="language-java"><span class="kd">public</span> <span class="kd">interface</span> <span class="nc">TypeSerializerSnapshot</span><span class="o">&lt;</span><span class="n">T</span><span class="o">&gt;</span> <span class="o">{</span>
+<span class="err"></span>
+ <span class="kt">int</span> <span class="nf">getCurrentVersion</span><span class="o">();</span>
+<span class="err"></span>
+ <span class="kt">void</span> <span class="nf">writeSnapshot</span><span class="o">(</span><span class="n">DataOutputView</span> <span class="n">out</span><span class="o">)</span> <span class="kd">throws</span> <span class="n">IOException</span><span class="o">;</span>
+<span class="err"></span>
+ <span class="kt">void</span> <span class="nf">readSnapshot</span><span class="o">(</span>
+ <span class="kt">int</span> <span class="n">readVersion</span><span class="o">,</span>
+ <span class="n">DataInputView</span> <span class="n">in</span><span class="o">,</span>
+ <span class="n">ClassLoader</span> <span class="n">userCodeClassLoader</span><span class="o">)</span> <span class="kd">throws</span> <span class="n">IOException</span><span class="o">;</span>
+<span class="err"></span>
+ <span class="n">TypeSerializer</span><span class="o">&lt;</span><span class="n">T</span><span class="o">&gt;</span> <span class="nf">restoreSerializer</span><span class="o">();</span>
+<span class="err"></span>
+ <span class="n">TypeSerializerSchemaCompatibility</span><span class="o">&lt;</span><span class="n">T</span><span class="o">&gt;</span> <span class="nf">resolveSchemaCompatibility</span><span class="o">(</span>
+ <span class="n">TypeSerializer</span><span class="o">&lt;</span><span class="n">T</span><span class="o">&gt;</span> <span class="n">newSerializer</span><span class="o">);</span>
+<span class="o">}</span></code></pre></div>
+
+<h2 id="implementing-apache-avro-serialization-in-flink">Implementing Apache Avro Serialization in Flink</h2>
+
+<p>Apache Avro is a data serialization format that has very well-defined schema migration semantics and supports both reader and writer schemas. During normal Flink execution the reader and writer schemas will be the same. However, when upgrading an application they may be different and with schema evolution, Flink will be able to migrate objects with their schemas.</p>
+
+<div class="highlight"><pre><code class="language-java"><span class="kd">public</span> <span class="kd">class</span> <span class="nc">AvroSerializerSnapshot</span><span class="o">&lt;</span><span class="n">T</span><span class="o">&gt;</span> <span class="kd">implements</span> <span class= [...]
+ <span class="kd">private</span> <span class="n">Schema</span> <span class="n">runtimeSchema</span><span class="o">;</span>
+ <span class="kd">private</span> <span class="n">Schema</span> <span class="n">previousSchema</span><span class="o">;</span>
+<span class="err"></span>
+ <span class="nd">@SuppressWarnings</span><span class="o">(</span><span class="s">&quot;WeakerAccess&quot;</span><span class="o">)</span>
+ <span class="kd">public</span> <span class="nf">AvroSerializerSnapshot</span><span class="o">()</span> <span class="o">{</span> <span class="o">}</span>
+<span class="err"></span>
+ <span class="n">AvroSerializerSnapshot</span><span class="o">(</span><span class="n">Schema</span> <span class="n">schema</span><span class="o">)</span> <span class="o">{</span>
+ <span class="k">this</span><span class="o">.</span><span class="na">runtimeSchema</span> <span class="o">=</span> <span class="n">schema</span><span class="o">;</span>
+ <span class="o">}</span></code></pre></div>
+
+<p>This is a sketch of our Avro serializer. It uses the provided schemas and delegates to Apache Avro for all (de)-serialization. Let’s take a look at one possible implementation of a <code>TypeSerializerSnapshot</code> that supports schema migration for Avro.</p>
+
+<h1 id="writing-out-the-snapshot">Writing out the snapshot</h1>
+
+<p>When serializing out the snapshot, the snapshot configuration will write two pieces of information; the current snapshot configuration version and the serializer configuration.</p>
+
+<div class="highlight"><pre><code class="language-java"> <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="kt">int</span> <span class="nf">getCurrentVersion</span><span class="o">()</span> <span class="o">{</span>
+ <span class="k">return</span> <span class="mi">1</span><span class="o">;</span>
+ <span class="o">}</span>
+<span class="err"></span>
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="nf">writeSnapshot</span><span class="o">(</span><span class="n">DataOutputView</span> <span class="n">out</span><span class="o">)</span> <span class="kd">throws</span> <span class="n">IOException</span> <span class=& [...]
+ <span class="n">out</span><span class="o">.</span><span class="na">writeUTF</span><span class="o">(</span><span class="n">runtimeSchema</span><span class="o">.</span><span class="na">toString</span><span class="o">(</span><span class="kc">false</span><span class="o">));< [...]
+ <span class="o">}</span></code></pre></div>
+
+<p>The version is used to version the snapshot configuration object itself while the <code>writeSnapshot</code> method writes out all the information we need to understand the current format; the runtime schema.</p>
+
+<div class="highlight"><pre><code class="language-java"> <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="nf">readSnapshot</span><span class="o">(</span>
+ <span class="kt">int</span> <span class="n">readVersion</span><span class="o">,</span>
+ <span class="n">DataInputView</span> <span class="n">in</span><span class="o">,</span>
+ <span class="n">ClassLoader</span> <span class="n">userCodeClassLoader</span><span class="o">)</span> <span class="kd">throws</span> <span class="n">IOException</span> <span class="o">{</span>
+
+ <span class="k">assert</span> <span class="n">readVersion</span> <span class="o">==</span> <span class="mi">1</span><span class="o">;</span>
+ <span class="kd">final</span> <span class="n">String</span> <span class="n">previousSchemaDefinition</span> <span class="o">=</span> <span class="n">in</span><span class="o">.</span><span class="na">readUTF</span><span class="o">();</span>
+ <span class="k">this</span><span class="o">.</span><span class="na">previousSchema</span> <span class="o">=</span> <span class="n">parseAvroSchema</span><span class="o">(</span><span class="n">previousSchemaDefinition</span><span class="o">);</span>
+ <span class="k">this</span><span class="o">.</span><span class="na">runtimeType</span> <span class="o">=</span> <span class="n">findClassOrFallbackToGeneric</span><span class="o">(</span>
+ <span class="n">userCodeClassLoader</span><span class="o">,</span>
+ <span class="n">previousSchema</span><span class="o">.</span><span class="na">getFullName</span><span class="o">());</span>
+<span class="err"></span>
+ <span class="k">this</span><span class="o">.</span><span class="na">runtimeSchema</span> <span class="o">=</span> <span class="n">tryExtractAvroSchema</span><span class="o">(</span><span class="n">userCodeClassLoader</span><span class="o">,</span> <span class="n">runtimeType</span><span [...]
+ <span class="o">}</span></code></pre></div>
+<p>Now when Flink restores it is able to read back in the writer schema used to serialize the data. The current runtime schema is discovered on the class path using some Java reflection magic.</p>
+
+<p>Once we have both of these we can compare them for compatibility. Perhaps nothing has changed and the schemas are compatible as is.</p>
+
+<div class="highlight"><pre><code class="language-java"> <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="n">TypeSerializerSchemaCompatibility</span><span class="o">&lt;</span><span class="n">T</span><span class="o">&gt;</span> <span class="nf">resolveSchemaCompatibility</span><span class="o">(</span>
+ <span class="n">TypeSerializer</span><span class="o">&lt;</span><span class="n">T</span><span class="o">&gt;</span> <span class="n">newSerializer</span><span class="o">)</span> <span class="o">{</span>
+<span class="err"></span>
+ <span class="k">if</span> <span class="o">(!(</span><span class="n">newSerializer</span> <span class="k">instanceof</span> <span class="n">AvroSerializer</span><span class="o">))</span> <span class="o">{</span>
+ <span class="k">return</span> <span class="n">TypeSerializerSchemaCompatibility</span><span class="o">.</span><span class="na">incompatible</span><span class="o">();</span>
+ <span class="o">}</span>
+<span class="err"></span>
+ <span class="k">if</span> <span class="o">(</span><span class="n">Objects</span><span class="o">.</span><span class="na">equals</span><span class="o">(</span><span class="n">previousSchema</span><span class="o">,</span> <span class="n">runtimeSchema</span><span class="o">)) [...]
+ <span class="k">return</span> <span class="n">TypeSerializerSchemaCompatibility</span><span class="o">.</span><span class="na">compatibleAsIs</span><span class="o">();</span>
+ <span class="o">}</span></code></pre></div>
+
+<p>Otherwise, the schemas are compared using Avro’s compatibility checks and they may either be compatible with a migration or incompatible.</p>
+
+<div class="highlight"><pre><code class="language-java"> <span class="kd">final</span> <span class="n">SchemaPairCompatibility</span> <span class="n">compatibility</span> <span class="o">=</span> <span class="n">SchemaCompatibility</span>
+ <span class="o">.</span><span class="na">checkReaderWriterCompatibility</span><span class="o">(</span><span class="n">previousSchema</span><span class="o">,</span> <span class="n">runtimeSchema</span><span class="o">);</span>
+<span class="err"></span>
+ <span class="k">return</span> <span class="nf">avroCompatibilityToFlinkCompatibility</span><span class="o">(</span><span class="n">compatibility</span><span class="o">);</span>
+ <span class="o">}</span></code></pre></div>
+
+<p>If they are compatible with migration then Flink will restore a new serializer that can read the old schema and deserialize into the new runtime type which is in effect a migration.</p>
+
+<div class="highlight"><pre><code class="language-java"> <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="n">TypeSerializer</span><span class="o">&lt;</span><span class="n">T</span><span class="o">&gt;</span> <span class="nf">restoreSerializer</span><span class="o">()</span> <span class="o">{</span>
+ <span class="k">if</span> <span class="o">(</span><span class="n">previousSchema</span> <span class="o">!=</span> <span class="kc">null</span><span class="o">)</span> <span class="o">{</span>
+ <span class="k">return</span> <span class="k">new</span> <span class="n">AvroSerializer</span><span class="o">&lt;&gt;(</span><span class="n">runtimeType</span><span class="o">,</span> <span class="n">runtimeSchema</span><span class="o">,</span> <span class="n">previousSchema</span& [...]
+ <span class="o">}</span> <span class="k">else</span> <span class="o">{</span>
+ <span class="k">return</span> <span class="k">new</span> <span class="n">AvroSerializer</span><span class="o">&lt;&gt;(</span><span class="n">runtimeType</span><span class="o">,</span> <span class="n">runtimeSchema</span><span class="o">,</span> <span class="n">runtimeSchema</span&g [...]
+ <span class="o">}</span>
+ <span class="o">}</span>
+<span class="o">}</span></code></pre></div>
+
+<h1 id="the-state-processor-api-reading-writing-and-modifying-flink-state">The State Processor API: Reading, writing and modifying Flink state</h1>
+
+<p>The State Processor API allows reading from and writing to Flink savepoints. Some of the interesting use cases it can be used for are:</p>
+
+<ul>
+ <li>Analyzing state for interesting patterns</li>
+ <li>Troubleshooting or auditing jobs by checking for state discrepancies</li>
+ <li>Bootstrapping state for new applications</li>
+ <li>Modifying savepoints such as:
+ <ul>
+ <li>Changing the maximum parallelism of a savepoint after deploying a Flink job</li>
+ <li>Introducing breaking schema updates to a Flink application</li>
+ <li>Correcting invalid state in a Flink savepoint</li>
+ </ul>
+ </li>
+</ul>
-In a [previous blog post](https://flink.apache.org/feature/2019/09/13/state-processor-api.html), we discussed the State Processor API in detail, the community’s motivation behind introducing the feature in Flink 1.9, what you can use the API for and how you can use it. Essentially, the State Processor API is based around a relational model of mapping your Flink job state to a database, as illustrated in the diagram below. We encourage you to [read the previous story](https://flink.apache [...]
+<p>In a <a href="https://flink.apache.org/feature/2019/09/13/state-processor-api.html">previous blog post</a>, we discussed the State Processor API in detail, the community’s motivation behind introducing the feature in Flink 1.9, what you can use the API for and how you can use it. Essentially, the State Processor API is based around a relational model of mapping your Flink job state to a database, as illustrated in the diagram below. We encourage you to < [...]
-* Reading Keyed and Operator State with the State Processor API and
-* Writing and Bootstrapping Keyed and Operator State with the State Processor API
+<ul>
+ <li>Reading Keyed and Operator State with the State Processor API and</li>
+ <li>Writing and Bootstrapping Keyed and Operator State with the State Processor API</li>
+</ul>
-Stay tuned for more details and guidance around this feature of Flink.
+<p>Stay tuned for more details and guidance around this feature of Flink.</p>
<center>
-<img src="{{ site.baseurl }}/img/blog/2020-01-29-state-unlocked-interacting-with-state-in-apache-flink/managing-state-in-flink-state-processor-api-visual-1.png" width="600px" alt="State Processor API in Apache Flink"/>
+<img src="/img/blog/2020-01-29-state-unlocked-interacting-with-state-in-apache-flink/managing-state-in-flink-state-processor-api-visual-1.png" width="600px" alt="State Processor API in Apache Flink" />
</center>
-<br>
+<p><br /></p>
<center>
-<img src="{{ site.baseurl }}/img/blog/2020-01-29-state-unlocked-interacting-with-state-in-apache-flink/managing-state-in-flink-state-processor-api-visual-2.png" width="600px" alt="State Processor API in Apache Flink"/>
-</center>
-<br>
-
-# Looking ahead: More ways to interact with State in Flink
-
-There is a lot of discussion happening in the community related to extending the way Flink developers interact with state in their Flink applications. Regarding the State Processor API, some thoughts revolve around further broadening the API’s scope beyond its current ability to read from and write to both keyed and operator state. In upcoming releases, the State processor API will be extended to support both reading from and writing to windows and have a first-class integration with Fli [...]
+<img src="/img/blog/2020-01-29-state-unlocked-interacting-with-state-in-apache-flink/managing-state-in-flink-state-processor-api-visual-2.png" width="600px" alt="State Processor API in Apache Flink" />
+</center>
+<p><br /></p>
-Beyond widening the scope of the State Processor API, the Flink community is discussing a few additional ways to improve the way developers interact with state in Flink. One of them is the proposal for a Unified Savepoint Format ([FLIP-41](https://cwiki.apache.org/confluence/display/FLINK/FLIP-41%3A+Unify+Binary+format+for+Keyed+State)) for all keyed state backends. Such improvement aims at introducing a unified binary format across all savepoints in all keyed state backends, something t [...]
+<h1 id="looking-ahead-more-ways-to-interact-with-state-in-flink">Looking ahead: More ways to interact with State in Flink</h1>
-The community is also discussing the ability to have upgradability dry runs in upcoming Flink releases. Having such functionality in Flink allows developers to detect incompatible updates offline without the need of starting a new Flink job from scratch. For example, Flink users will be able to uncover topology or schema incompatibilities upon upgrading a Flink job, without having to load the state back to a running Flink job in the first place. Additionally, with upgradability dry runs [...]
+<p>There is a lot of discussion happening in the community related to extending the way Flink developers interact with state in their Flink applications. Regarding the State Processor API, some thoughts revolve around further broadening the API’s scope beyond its current ability to read from and write to both keyed and operator state. In upcoming releases, the State processor API will be extended to support both reading from and writing to windows and have a first-class integration [...]
-With all the exciting new functionality added in Flink 1.9 as well as some solid ideas and discussions around bringing state in Flink to the next level, the community is committed to making state in Apache Flink a fundamental element of the framework, something that is ever-present across versions and upgrades of your application and a component that is a true first-class citizen in Apache Flink. We encourage you to sign up to the [mailing list](https://flink.apache.org/community.html) [...]
+<p>Beyond widening the scope of the State Processor API, the Flink community is discussing a few additional ways to improve the way developers interact with state in Flink. One of them is the proposal for a Unified Savepoint Format (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-41%3A+Unify+Binary+format+for+Keyed+State">FLIP-41</a>) for all keyed state backends. Such improvement aims at introducing a unified binary format across all savepoint [...]
+
+<p>The community is also discussing the ability to have upgradability dry runs in upcoming Flink releases. Having such functionality in Flink allows developers to detect incompatible updates offline without the need of starting a new Flink job from scratch. For example, Flink users will be able to uncover topology or schema incompatibilities upon upgrading a Flink job, without having to load the state back to a running Flink job in the first place. Additionally, with upgradability [...]
+
+<p>With all the exciting new functionality added in Flink 1.9 as well as some solid ideas and discussions around bringing state in Flink to the next level, the community is committed to making state in Apache Flink a fundamental element of the framework, something that is ever-present across versions and upgrades of your application and a component that is a true first-class citizen in Apache Flink. We encourage you to sign up to the <a href="https://flink.apache.org/commu [...]
</description>
-<pubDate>Wed, 29 Jan 2020 12:00:00 +0000</pubDate>
+<pubDate>Wed, 29 Jan 2020 13:00:00 +0100</pubDate>
<link>https://flink.apache.org/news/2020/01/29/state-unlocked-interacting-with-state-in-apache-flink.html</link>
<guid isPermaLink="true">/news/2020/01/29/state-unlocked-interacting-with-state-in-apache-flink.html</guid>
</item>
<item>
<title>Advanced Flink Application Patterns Vol.1: Case Study of a Fraud Detection System</title>
-<description>In this series of blog posts you will learn about three powerful Flink patterns for building streaming applications:
+<description><p>In this series of blog posts you will learn about three powerful Flink patterns for building streaming applications:</p>
- - Dynamic updates of application logic
- - Dynamic data partitioning (shuffle), controlled at runtime
- - Low latency alerting based on custom windowing logic (without using the window API)
-
-These patterns expand the possibilities of what is achievable with statically defined data flows and provide the building blocks to fulfill complex business requirements.
+<ul>
+ <li>Dynamic updates of application logic</li>
+ <li>Dynamic data partitioning (shuffle), controlled at runtime</li>
+ <li>Low latency alerting based on custom windowing logic (without using the window API)</li>
+</ul>
-**Dynamic updates of application logic** allow Flink jobs to change at runtime, without downtime from stopping and resubmitting the code.
-<br>
-**Dynamic data partitioning** provides the ability to change how events are distributed and grouped by Flink at runtime. Such functionality often becomes a natural requirement when building jobs with dynamically reconfigurable application logic.
-<br>
-**Custom window management** demonstrates how you can utilize the low level [process function API](https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/process_function.html), when the native [window API](https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/windows.html) is not exactly matching your requirements. Specifically, you will learn how to implement low latency alerting on windows and how to limit state growth with timers.
+<p>These patterns expand the possibilities of what is achievable with statically defined data flows and provide the building blocks to fulfill complex business requirements.</p>
-These patterns build on top of core Flink functionality, however, they might not be immediately apparent from the framework's documentation as explaining and presenting the motivation behind them is not always trivial without a concrete use case. That is why we will showcase these patterns with a practical example that offers a real-world usage scenario for Apache Flink — a _Fraud Detection_ engine.
-We hope that this series will place these powerful approaches into your tool belt and enable you to take on new and exciting tasks.
+<p><strong>Dynamic updates of application logic</strong> allow Flink jobs to change at runtime, without downtime from stopping and resubmitting the code.<br />
+<br />
+<strong>Dynamic data partitioning</strong> provides the ability to change how events are distributed and grouped by Flink at runtime. Such functionality often becomes a natural requirement when building jobs with dynamically reconfigurable application logic.<br />
+<br />
+<strong>Custom window management</strong> demonstrates how you can utilize the low level <a href="https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/process_function.html">process function API</a>, when the native <a href="https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/windows.html">window API</a> is not exactly matching your requirements. Specifically, you will learn how to impl [...]
-In the first blog post of the series we will look at the high-level architecture of the demo application, describe its components and their interactions. We will then deep dive into the implementation details of the first pattern in the series - **dynamic data partitioning**.
+<p>These patterns build on top of core Flink functionality, however, they might not be immediately apparent from the framework’s documentation as explaining and presenting the motivation behind them is not always trivial without a concrete use case. That is why we will showcase these patterns with a practical example that offers a real-world usage scenario for Apache Flink — a <em>Fraud Detection</em> engine.
+We hope that this series will place these powerful approaches into your tool belt and enable you to take on new and exciting tasks.</p>
+<p>In the first blog post of the series we will look at the high-level architecture of the demo application, describe its components and their interactions. We will then deep dive into the implementation details of the first pattern in the series - <strong>dynamic data partitioning</strong>.</p>
-You will be able to run the full Fraud Detection Demo application locally and look into the details of the implementation by using the accompanying GitHub repository.
+<p>You will be able to run the full Fraud Detection Demo application locally and look into the details of the implementation by using the accompanying GitHub repository.</p>
-### Fraud Detection Demo
+<h3 id="fraud-detection-demo">Fraud Detection Demo</h3>
-The full source code for our fraud detection demo is open source and available online. To run it locally, check out the following repository and follow the steps in the README:
+<p>The full source code for our fraud detection demo is open source and available online. To run it locally, check out the following repository and follow the steps in the README:</p>
-[https://github.com/afedulov/fraud-detection-demo](https://github.com/afedulov/fraud-detection-demo)
+<p><a href="https://github.com/afedulov/fraud-detection-demo">https://github.com/afedulov/fraud-detection-demo</a></p>
-You will see the demo is a self-contained application - it only requires `docker` and `docker-compose` to be built from sources and includes the following components:
+<p>You will see the demo is a self-contained application - it only requires <code>docker</code> and <code>docker-compose</code> to be built from sources and includes the following components:</p>
- - Apache Kafka (message broker) with ZooKeeper
- - Apache Flink ([application cluster](https://ci.apache.org/projects/flink/flink-docs-stable/concepts/glossary.html#flink-application-cluster))
- - Fraud Detection Web App
+<ul>
+ <li>Apache Kafka (message broker) with ZooKeeper</li>
+ <li>Apache Flink (<a href="https://ci.apache.org/projects/flink/flink-docs-stable/concepts/glossary.html#flink-application-cluster">application cluster</a>)</li>
+ <li>Fraud Detection Web App</li>
+</ul>
-The high-level goal of the Fraud Detection engine is to consume a stream of financial transactions and evaluate them against a set of rules. These rules are subject to frequent changes and tweaks. In a real production system, it is important to be able to add and remove them at runtime, without incurring an expensive penalty of stopping and restarting the job.
+<p>The high-level goal of the Fraud Detection engine is to consume a stream of financial transactions and evaluate them against a set of rules. These rules are subject to frequent changes and tweaks. In a real production system, it is important to be able to add and remove them at runtime, without incurring an expensive penalty of stopping and restarting the job.</p>
-When you navigate to the demo URL in your browser, you will be presented with the following UI:
+<p>When you navigate to the demo URL in your browser, you will be presented with the following UI:</p>
- <center>
- <img src="{{ site.baseurl }}/img/blog/2019-11-19-demo-fraud-detection/ui.png" width="800px" alt="Figure 1: Demo UI"/>
- <br/>
+<center>
+ <img src="/img/blog/2019-11-19-demo-fraud-detection/ui.png" width="800px" alt="Figure 1: Demo UI" />
+ <br />
<i><small>Figure 1: Fraud Detection Demo UI</small></i>
</center>
- <br/>
+<p><br /></p>
-On the left side, you can see a visual representation of financial transactions flowing through the system after you click the "Start" button. The slider at the top allows you to control the number of generated transactions per second. The middle section is devoted to managing the rules evaluated by Flink. From here, you can create new rules as well as issue control commands, such as clearing Flink's state.
+<p>On the left side, you can see a visual representation of financial transactions flowing through the system after you click the “Start” button. The slider at the top allows you to control the number of generated transactions per second. The middle section is devoted to managing the rules evaluated by Flink. From here, you can create new rules as well as issue control commands, such as clearing Flink’s state.</p>
-The demo out-of-the-box comes with a set of predefined sample rules. You can click the _Start_ button and, after some time, will observe alerts displayed in the right section of the UI. These alerts are the result of Flink evaluating the generated transactions stream against the predefined rules.
+<p>The demo out-of-the-box comes with a set of predefined sample rules. You can click the <em>Start</em> button and, after some time, will observe alerts displayed in the right section of the UI. These alerts are the result of Flink evaluating the generated transactions stream against the predefined rules.</p>
- Our sample fraud detection system consists of three main components:
+<p>Our sample fraud detection system consists of three main components:</p>
- 1. Frontend (React)
- 1. Backend (SpringBoot)
- 1. Fraud Detection application (Apache Flink)
+<ol>
+ <li>Frontend (React)</li>
+ <li>Backend (SpringBoot)</li>
+ <li>Fraud Detection application (Apache Flink)</li>
+</ol>
-Interactions between the main elements are depicted in _Figure 2_.
+<p>Interactions between the main elements are depicted in <em>Figure 2</em>.</p>
- <center>
- <img src="{{ site.baseurl }}/img/blog/2019-11-19-demo-fraud-detection/architecture.png" width="800px" alt="Figure 2: Demo Components"/>
- <br/>
+<center>
+ <img src="/img/blog/2019-11-19-demo-fraud-detection/architecture.png" width="800px" alt="Figure 2: Demo Components" />
+ <br />
<i><small>Figure 2: Fraud Detection Demo Components</small></i>
</center>
- <br/>
+<p><br /></p>
- The Backend exposes a REST API to the Frontend for creating/deleting rules as well as issuing control commands for managing the demo execution. It then relays those Frontend actions to Flink by sending them via a "Control" Kafka topic. The Backend additionally includes a _Transactions Generator_ component, which sends an emulated stream of money transfer events to Flink via a separate "Transactions" topic. Alerts generated by Flink are consumed by the Backend from &q [...]
+<p>The Backend exposes a REST API to the Frontend for creating/deleting rules as well as issuing control commands for managing the demo execution. It then relays those Frontend actions to Flink by sending them via a “Control” Kafka topic. The Backend additionally includes a <em>Transactions Generator</em> component, which sends an emulated stream of money transfer events to Flink via a separate “Transactions” topic. Alerts generated by Flink are consumed by the Backend [...]
-Now that you are familiar with the overall layout and the goal of our Fraud Detection engine, let's now go into the details of what is required to implement such a system.
+<p>Now that you are familiar with the overall layout and the goal of our Fraud Detection engine, let’s now go into the details of what is required to implement such a system.</p>
-### Dynamic Data Partitioning
+<h3 id="dynamic-data-partitioning">Dynamic Data Partitioning</h3>
-The first pattern we will look into is Dynamic Data Partitioning.
+<p>The first pattern we will look into is Dynamic Data Partitioning.</p>
-If you have used Flink's DataStream API in the past, you are undoubtedly familiar with the **keyBy** method. Keying a stream shuffles all the records such that elements with the same key are assigned to the same partition. This means all records with the same key are processed by the same physical instance of the next operator.
+<p>If you have used Flink’s DataStream API in the past, you are undoubtedly familiar with the <strong>keyBy</strong> method. Keying a stream shuffles all the records such that elements with the same key are assigned to the same partition. This means all records with the same key are processed by the same physical instance of the next operator.</p>
-In a typical streaming application, the choice of key is fixed, determined by some static field within the elements. For instance, when building a simple window-based aggregation of a stream of transactions, we might always group by the transactions account id.
+<p>In a typical streaming application, the choice of key is fixed, determined by some static field within the elements. For instance, when building a simple window-based aggregation of a stream of transactions, we might always group by the transactions account id.</p>
-```java
-DataStream<Transaction> input = // [...]
-DataStream<...> windowed = input
- .keyBy(Transaction::getAccountId)
- .window(/*window specification*/);
-```
+<div class="highlight"><pre><code class="language-java"><span class="n">DataStream</span><span class="o">&lt;</span><span class="n">Transaction</span><span class="o">&gt;</span> <span class="n">input</span> <span class="o">=</span> <span class="c1">// [...]</span>
+<span class="n">DataStream</span><span class="o">&lt;...&gt;</span> <span class="n">windowed</span> <span class="o">=</span> <span class="n">input</span>
+ <span class="o">.</span><span class="na">keyBy</span><span class="o">(</span><span class="nl">Transaction:</span><span class="o">:</span><span class="n">getAccountId</span><span class="o">)</span>
+ <span class="o">.</span><span class="na">window</span><span class="o">(</span><span class="cm">/*window specification*/</span><span class="o">);</span></code></pre></div>
-This approach is the main building block for achieving horizontal scalability in a wide range of use cases. However, in the case of an application striving to provide flexibility in business logic at runtime, this is not enough.
-To understand why this is the case, let us start with articulating a realistic sample rule definition for our fraud detection system in the form of a functional requirement:
+<p>This approach is the main building block for achieving horizontal scalability in a wide range of use cases. However, in the case of an application striving to provide flexibility in business logic at runtime, this is not enough.
+To understand why this is the case, let us start with articulating a realistic sample rule definition for our fraud detection system in the form of a functional requirement:</p>
-*"Whenever the **sum** of the accumulated **payment amount** from the same **payer** to the same **beneficiary** within the **duration of a week** is **greater** than **1 000 000 $** - fire an alert."*
+<p><em>“Whenever the <strong>sum</strong> of the accumulated <strong>payment amount</strong> from the same <strong>payer</strong> to the same <strong>beneficiary</strong> within the <strong>duration of a week</strong> is <strong>greater</strong> than <strong>1 000 000 $</strong> - fire an alert.”</em></p>
-In this formulation we can spot a number of parameters that we would like to be able to specify in a newly-submitted rule and possibly even later modify or tweak at runtime:
+<p>In this formulation we can spot a number of parameters that we would like to be able to specify in a newly-submitted rule and possibly even later modify or tweak at runtime:</p>
-1. Aggregation field (payment amount)
-1. Grouping fields (payer + beneficiary)
-1. Aggregation function (sum)
-1. Window duration (1 week)
-1. Limit (1 000 000)
-1. Limit operator (greater)
+<ol>
+ <li>Aggregation field (payment amount)</li>
+ <li>Grouping fields (payer + beneficiary)</li>
+ <li>Aggregation function (sum)</li>
+ <li>Window duration (1 week)</li>
+ <li>Limit (1 000 000)</li>
+ <li>Limit operator (greater)</li>
+</ol>
-Accordingly, we will use the following simple JSON format to define the aforementioned parameters:
+<p>Accordingly, we will use the following simple JSON format to define the aforementioned parameters:</p>
-```json
-{
- "ruleId": 1,
- "ruleState": "ACTIVE",
- "groupingKeyNames": ["payerId", "beneficiaryId"],
- "aggregateFieldName": "paymentAmount",
- "aggregatorFunctionType": "SUM",
- "limitOperatorType": "GREATER",
- "limit": 1000000,
- "windowMinutes": 10080
-}
-```
+<div class="highlight"><pre><code class="language-json"><span class="p">{</span>
+ <span class="nt">&quot;ruleId&quot;</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
+ <span class="nt">&quot;ruleState&quot;</span><span class="p">:</span> <span class="s2">&quot;ACTIVE&quot;</span><span class="p">,</span>
+ <span class="nt">&quot;groupingKeyNames&quot;</span><span class="p">:</span> <span class="p">[</span><span class="s2">&quot;payerId&quot;</span><span class="p">,</span> <span class="s2">&quot;beneficiaryId&quot;</span><span class="p">],</span>
+ <span class="nt">&quot;aggregateFieldName&quot;</span><span class="p">:</span> <span class="s2">&quot;paymentAmount&quot;</span><span class="p">,</span>
+ <span class="nt">&quot;aggregatorFunctionType&quot;</span><span class="p">:</span> <span class="s2">&quot;SUM&quot;</span><span class="p">,</span>
+ <span class="nt">&quot;limitOperatorType&quot;</span><span class="p">:</span> <span class="s2">&quot;GREATER&quot;</span><span class="p">,</span>
+ <span class="nt">&quot;limit&quot;</span><span class="p">:</span> <span class="mi">1000000</span><span class="p">,</span>
+ <span class="nt">&quot;windowMinutes&quot;</span><span class="p">:</span> <span class="mi">10080</span>
+<span class="p">}</span></code></pre></div>
-At this point, it is important to understand that **`groupingKeyNames`** determine the actual physical grouping of events - all Transactions with the same values of specified parameters (e.g. _payer #25 -> beneficiary #12_) have to be aggregated in the same physical instance of the evaluating operator. Naturally, the process of distributing data in such a way in Flink's API is realised by a `keyBy()` function.
+<p>At this point, it is important to understand that <strong><code>groupingKeyNames</code></strong> determine the actual physical grouping of events - all Transactions with the same values of specified parameters (e.g. <em>payer #25 -&gt; beneficiary #12</em>) have to be aggregated in the same physical instance of the evaluating operator. Naturally, the process of distributing data in such a way in Flink’s API is realised by a <code>key [...]
-Most examples in Flink's `keyBy()`[documentation](https://ci.apache.org/projects/flink/flink-docs-stable/dev/api_concepts.html#define-keys-using-field-expressions) use a hard-coded `KeySelector`, which extracts specific fixed events' fields. However, to support the desired flexibility, we have to extract them in a more dynamic fashion based on the specifications of the rules. For this, we will have to use one additional operator that prepares every event for dispatching to a corr [...]
+<p>Most examples in Flink’s <code>keyBy()</code><a href="https://ci.apache.org/projects/flink/flink-docs-stable/dev/api_concepts.html#define-keys-using-field-expressions">documentation</a> use a hard-coded <code>KeySelector</code>, which extracts specific fixed events’ fields. However, to support the desired flexibility, we have to extract them in a more dynamic fashion based on the specifications of the rules. For this, we will have to [...]
-On a high level, our main processing pipeline looks like this:
+<p>On a high level, our main processing pipeline looks like this:</p>
-```java
-DataStream<Alert> alerts =
- transactions
- .process(new DynamicKeyFunction())
- .keyBy(/* some key selector */);
- .process(/* actual calculations and alerting */)
-```
+<div class="highlight"><pre><code class="language-java"><span class="n">DataStream</span><span class="o">&lt;</span><span class="n">Alert</span><span class="o">&gt;</span> <span class="n">alerts</span> <span class="o">=</span>
+ <span class="n">transactions</span>
+ <span class="o">.</span><span class="na">process</span><span class="o">(</span><span class="k">new</span> <span class="nf">DynamicKeyFunction</span><span class="o">())</span>
+ <span class="o">.</span><span class="na">keyBy</span><span class="o">(</span><span class="cm">/* some key selector */</span><span class="o">);</span>
+ <span class="o">.</span><span class="na">process</span><span class="o">(</span><span class="cm">/* actual calculations and alerting */</span><span class="o">)</span></code></pre></div>
-We have previously established that each rule defines a **`groupingKeyNames`** parameter that specifies which combination of fields will be used for the incoming events' grouping. Each rule might use an arbitrary combination of these fields. At the same time, every incoming event potentially needs to be evaluated against multiple rules. This implies that events might simultaneously need to be present at multiple parallel instances of evaluating operators that correspond to different [...]
+<p>We have previously established that each rule defines a <strong><code>groupingKeyNames</code></strong> parameter that specifies which combination of fields will be used for the incoming events’ grouping. Each rule might use an arbitrary combination of these fields. At the same time, every incoming event potentially needs to be evaluated against multiple rules. This implies that events might simultaneously need to be present at multiple parallel instances [...]
<center>
-<img src="{{ site.baseurl }}/img/blog/2019-11-19-demo-fraud-detection/shuffle_function_1.png" width="800px" alt="Figure 3: Forking events with Dynamic Key Function"/>
-<br/>
+<img src="/img/blog/2019-11-19-demo-fraud-detection/shuffle_function_1.png" width="800px" alt="Figure 3: Forking events with Dynamic Key Function" />
+<br />
<i><small>Figure 3: Forking events with Dynamic Key Function</small></i>
</center>
-<br/>
+<p><br /></p>
- `DynamicKeyFunction` iterates over a set of defined rules and prepares every event to be processed by a `keyBy()` function by extracting the required grouping keys:
+<p><code>DynamicKeyFunction</code> iterates over a set of defined rules and prepares every event to be processed by a <code>keyBy()</code> function by extracting the required grouping keys:</p>
-```java
-public class DynamicKeyFunction
- extends ProcessFunction<Transaction, Keyed<Transaction, String, Integer>> {
- ...
- /* Simplified */
- List<Rule> rules = /* Rules that are initialized somehow.
- Details will be discussed in a future blog post. */;
+<div class="highlight"><pre><code class="language-java"><span class="kd">public</span> <span class="kd">class</span> <span class="nc">DynamicKeyFunction</span>
+ <span class="kd">extends</span> <span class="n">ProcessFunction</span><span class="o">&lt;</span><span class="n">Transaction</span><span class="o">,</span> <span class="n">Keyed</span><span class="o">&lt;</span><span class="n">Transaction</span><span class="o">,</span> <span cla [...]
+ <span class="o">...</span>
+ <span class="cm">/* Simplified */</span>
+ <span class="n">List</span><span class="o">&lt;</span><span class="n">Rule</span><span class="o">&gt;</span> <span class="n">rules</span> <span class="o">=</span> <span class="cm">/* Rules that are initialized somehow.</span>
+<span class="cm"> Details will be discussed in a future blog post. */</span><span class="o">;</span>
- @Override
- public void processElement(
- Transaction event,
- Context ctx,
- Collector<Keyed<Transaction, String, Integer>> out) {
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="nf">processElement</span><span class="o">(</span>
+ <span class="n">Transaction</span> <span class="n">event</span><span class="o">,</span>
+ <span class="n">Context</span> <span class="n">ctx</span><span class="o">,</span>
+ <span class="n">Collector</span><span class="o">&lt;</span><span class="n">Keyed</span><span class="o">&lt;</span><span class="n">Transaction</span><span class="o">,</span> <span class="n">String</span><span class="o">,</span> <span class="n">Integer</span><span class="o& [...]
- for (Rule rule :rules) {
- out.collect(
- new Keyed<>(
- event,
- KeysExtractor.getKey(rule.getGroupingKeyNames(), event),
- rule.getRuleId()));
- }
- }
- ...
-}
-```
- `KeysExtractor.getKey()` uses reflection to extract the required values of `groupingKeyNames` fields from events and combines them as a single concatenated String key, e.g `"{payerId=25;beneficiaryId=12}"`. Flink will calculate the hash of this key and assign the processing of this particular combination to a specific server in the cluster. This will allow tracking all transactions between _payer #25_ and _beneficiary #12_ and evaluating defined rules within the desired time window.
+ <span class="k">for</span> <span class="o">(</span><span class="n">Rule</span> <span class="n">rule</span> <span class="o">:</span><span class="n">rules</span><span class="o">)</span> <span class="o">{</span>
+ <span class="n">out</span><span class="o">.</span><span class="na">collect</span><span class="o">(</span>
+ <span class="k">new</span> <span class="n">Keyed</span><span class="o">&lt;&gt;(</span>
+ <span class="n">event</span><span class="o">,</span>
+ <span class="n">KeysExtractor</span><span class="o">.</span><span class="na">getKey</span><span class="o">(</span><span class="n">rule</span><span class="o">.</span><span class="na">getGroupingKeyNames</span><span class="o">(),</span> <span class="n">event</span><span class=& [...]
+ <span class="n">rule</span><span class="o">.</span><span class="na">getRuleId</span><span class="o">()));</span>
+ <span class="o">}</span>
+ <span class="o">}</span>
+ <span class="o">...</span>
+<span class="o">}</span></code></pre></div>
+<p><code>KeysExtractor.getKey()</code> uses reflection to extract the required values of <code>groupingKeyNames</code> fields from events and combines them as a single concatenated String key, e.g <code>"{payerId=25;beneficiaryId=12}"</code>. Flink will calculate the hash of this key and assign the processing of this particular combination to a specific server in the cluster. This will allow tracking all transactions between <em>p [...]
-Notice that a wrapper class `Keyed` with the following signature was introduced as the output type of `DynamicKeyFunction`:
+<p>Notice that a wrapper class <code>Keyed</code> with the following signature was introduced as the output type of <code>DynamicKeyFunction</code>:</p>
-```java
-public class Keyed<IN, KEY, ID> {
- private IN wrapped;
- private KEY key;
- private ID id;
+<div class="highlight"><pre><code class="language-java"><span class="kd">public</span> <span class="kd">class</span> <span class="nc">Keyed</span><span class="o">&lt;</span><span class="n">IN</span><span class="o">,</span> <span class="n">KEY</span><span class="o">,</span> [...]
+ <span class="kd">private</span> <span class="n">IN</span> <span class="n">wrapped</span><span class="o">;</span>
+ <span class="kd">private</span> <span class="n">KEY</span> <span class="n">key</span><span class="o">;</span>
+ <span class="kd">private</span> <span class="n">ID</span> <span class="n">id</span><span class="o">;</span>
- ...
- public KEY getKey(){
- return key;
- }
-}
-```
+ <span class="o">...</span>
+ <span class="kd">public</span> <span class="n">KEY</span> <span class="nf">getKey</span><span class="o">(){</span>
+ <span class="k">return</span> <span class="n">key</span><span class="o">;</span>
+ <span class="o">}</span>
+<span class="o">}</span></code></pre></div>
-Fields of this POJO carry the following information: `wrapped` is the original transaction event, `key` is the result of using `KeysExtractor` and `id` is the ID of the Rule that caused the dispatch of the event (according to the rule-specific grouping logic).
+<p>Fields of this POJO carry the following information: <code>wrapped</code> is the original transaction event, <code>key</code> is the result of using <code>KeysExtractor</code> and <code>id</code> is the ID of the Rule that caused the dispatch of the event (according to the rule-specific grouping logic).</p>
-Events of this type will be the input to the `keyBy()` function in the main processing pipeline and allow the use of a simple lambda-expression as a [`KeySelector`](https://ci.apache.org/projects/flink/flink-docs-stable/dev/api_concepts.html#define-keys-using-key-selector-functions) for the final step of implementing dynamic data shuffle.
+<p>Events of this type will be the input to the <code>keyBy()</code> function in the main processing pipeline and allow the use of a simple lambda-expression as a <a href="https://ci.apache.org/projects/flink/flink-docs-stable/dev/api_concepts.html#define-keys-using-key-selector-functions"><code>KeySelector</code></a> for the final step of implementing dynamic data shuffle.</p>
-```java
-DataStream<Alert> alerts =
- transactions
- .process(new DynamicKeyFunction())
- .keyBy((keyed) -> keyed.getKey());
- .process(new DynamicAlertFunction())
-```
+<div class="highlight"><pre><code class="language-java"><span class="n">DataStream</span><span class="o">&lt;</span><span class="n">Alert</span><span class="o">&gt;</span> <span class="n">alerts</span> <span class="o">=</span>
+ <span class="n">transactions</span>
+ <span class="o">.</span><span class="na">process</span><span class="o">(</span><span class="k">new</span> <span class="nf">DynamicKeyFunction</span><span class="o">())</span>
+ <span class="o">.</span><span class="na">keyBy</span><span class="o">((</span><span class="n">keyed</span><span class="o">)</span> <span class="o">-&gt;</span> <span class="n">keyed</span><span class="o">.</span><span class="na">getKey</span><span class="o">());< [...]
+ <span class="o">.</span><span class="na">process</span><span class="o">(</span><span class="k">new</span> <span class="nf">DynamicAlertFunction</span><span class="o">())</span></code></pre></div>
-By applying `DynamicKeyFunction` we are implicitly copying events for performing parallel per-rule evaluation within a Flink cluster. By doing so, we achieve an important property - horizontal scalability of rules' processing. Our system will be capable of handling more rules by adding more servers to the cluster, i.e. increasing the parallelism. This property is achieved at the cost of data duplication, which might become an issue depending on the specific set of parameters, such as [...]
+<p>By applying <code>DynamicKeyFunction</code> we are implicitly copying events for performing parallel per-rule evaluation within a Flink cluster. By doing so, we achieve an important property - horizontal scalability of rules’ processing. Our system will be capable of handling more rules by adding more servers to the cluster, i.e. increasing the parallelism. This property is achieved at the cost of data duplication, which might become an issue depending on the specifi [...]
-### Summary:
+<h3 id="summary">Summary:</h3>
-In this blog post, we have discussed the motivation behind supporting dynamic, runtime changes to a Flink application by looking at a sample use case - a Fraud Detection engine. We have described the overall architecture and interactions between its components as well as provided references for building and running a demo Fraud Detection application in a dockerized setup. We then showed the details of implementing a **dynamic data partitioning pattern** as the first underlying building [...]
+<p>In this blog post, we have discussed the motivation behind supporting dynamic, runtime changes to a Flink application by looking at a sample use case - a Fraud Detection engine. We have described the overall architecture and interactions between its components as well as provided references for building and running a demo Fraud Detection application in a dockerized setup. We then showed the details of implementing a <strong>dynamic data partitioning pattern</strong> [...]
-To remain focused on describing the core mechanics of the pattern, we kept the complexity of the DSL and the underlying rules engine to a minimum. Going forward, it is easy to imagine adding extensions such as allowing more sophisticated rule definitions, including filtering of certain events, logical rules chaining, and other more advanced functionality.
+<p>To remain focused on describing the core mechanics of the pattern, we kept the complexity of the DSL and the underlying rules engine to a minimum. Going forward, it is easy to imagine adding extensions such as allowing more sophisticated rule definitions, including filtering of certain events, logical rules chaining, and other more advanced functionality.</p>
-In the second part of this series, we will describe how the rules make their way into the running Fraud Detection engine. Additionally, we will go over the implementation details of the main processing function of the pipeline - _DynamicAlertFunction()_.
+<p>In the second part of this series, we will describe how the rules make their way into the running Fraud Detection engine. Additionally, we will go over the implementation details of the main processing function of the pipeline - <em>DynamicAlertFunction()</em>.</p>
<center>
-<img src="{{ site.baseurl }}/img/blog/2019-11-19-demo-fraud-detection/end-to-end.png" width="800px" alt="Figure 4: End-to-end pipeline"/>
-<br/>
+<img src="/img/blog/2019-11-19-demo-fraud-detection/end-to-end.png" width="800px" alt="Figure 4: End-to-end pipeline" />
+<br />
<i><small>Figure 4: End-to-end pipeline</small></i>
</center>
-<br/>
+<p><br /></p>
-In the next article, we will see how Flink's broadcast streams can be utilized to help steer the processing within the Fraud Detection engine at runtime (Dynamic Application Updates pattern).
+<p>In the next article, we will see how Flink’s broadcast streams can be utilized to help steer the processing within the Fraud Detection engine at runtime (Dynamic Application Updates pattern).</p>
</description>
-<pubDate>Wed, 15 Jan 2020 12:00:00 +0000</pubDate>
+<pubDate>Wed, 15 Jan 2020 13:00:00 +0100</pubDate>
<link>https://flink.apache.org/news/2020/01/15/demo-fraud-detection.html</link>
<guid isPermaLink="true">/news/2020/01/15/demo-fraud-detection.html</guid>
</item>
<item>
<title>Apache Flink 1.8.3 Released</title>
-<description>The Apache Flink community released the third bugfix version of the Apache Flink 1.8 series.
+<description><p>The Apache Flink community released the third bugfix version of the Apache Flink 1.8 series.</p>
-This release includes 45 fixes and minor improvements for Flink 1.8.2. The list below includes a detailed list of all fixes and improvements.
+<p>This release includes 45 fixes and minor improvements for Flink 1.8.2. The list below includes a detailed list of all fixes and improvements.</p>
-We highly recommend all users to upgrade to Flink 1.8.3.
+<p>We highly recommend all users to upgrade to Flink 1.8.3.</p>
-Updated Maven dependencies:
+<p>Updated Maven dependencies:</p>
-```xml
-<dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-java</artifactId>
- <version>1.8.3</version>
-</dependency>
-<dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-streaming-java_2.11</artifactId>
- <version>1.8.3</version>
-</dependency>
-<dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-clients_2.11</artifactId>
- <version>1.8.3</version>
-</dependency>
-```
+<div class="highlight"><pre><code class="language-xml"><span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-java<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.8.3<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span>
+<span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-streaming-java_2.11<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.8.3<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span>
+<span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-clients_2.11<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.8.3<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span></code></pre></div>
-You can find the binaries on the updated [Downloads page]({{ site.baseurl }}/downloads.html).
+<p>You can find the binaries on the updated <a href="/downloads.html">Downloads page</a>.</p>
-List of resolved issues:
+<p>List of resolved issues:</p>
<h2> Sub-task
</h2>
<ul>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13723'>FLINK-13723</a>] - Use liquid-c for faster doc generation
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13723">FLINK-13723</a>] - Use liquid-c for faster doc generation
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13724'>FLINK-13724</a>] - Remove unnecessary whitespace from the docs&#39; sidenav
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13724">FLINK-13724</a>] - Remove unnecessary whitespace from the docs&#39; sidenav
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13725'>FLINK-13725</a>] - Use sassc for faster doc generation
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13725">FLINK-13725</a>] - Use sassc for faster doc generation
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13726'>FLINK-13726</a>] - Build docs with jekyll 4.0.0.pre.beta1
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13726">FLINK-13726</a>] - Build docs with jekyll 4.0.0.pre.beta1
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13791'>FLINK-13791</a>] - Speed up sidenav by using group_by
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13791">FLINK-13791</a>] - Speed up sidenav by using group_by
</li>
</ul>
-
+
<h2> Bug
</h2>
<ul>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-12342'>FLINK-12342</a>] - Yarn Resource Manager Acquires Too Many Containers
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12342">FLINK-12342</a>] - Yarn Resource Manager Acquires Too Many Containers
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13184'>FLINK-13184</a>] - Starting a TaskExecutor blocks the YarnResourceManager&#39;s main thread
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13184">FLINK-13184</a>] - Starting a TaskExecutor blocks the YarnResourceManager&#39;s main thread
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13728'>FLINK-13728</a>] - Fix wrong closing tag order in sidenav
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13728">FLINK-13728</a>] - Fix wrong closing tag order in sidenav
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13746'>FLINK-13746</a>] - Elasticsearch (v2.3.5) sink end-to-end test fails on Travis
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13746">FLINK-13746</a>] - Elasticsearch (v2.3.5) sink end-to-end test fails on Travis
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13749'>FLINK-13749</a>] - Make Flink client respect classloading policy
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13749">FLINK-13749</a>] - Make Flink client respect classloading policy
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13892'>FLINK-13892</a>] - HistoryServerTest failed on Travis
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13892">FLINK-13892</a>] - HistoryServerTest failed on Travis
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13936'>FLINK-13936</a>] - NOTICE-binary is outdated
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13936">FLINK-13936</a>] - NOTICE-binary is outdated
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13966'>FLINK-13966</a>] - Jar sorting in collect_license_files.sh is locale dependent
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13966">FLINK-13966</a>] - Jar sorting in collect_license_files.sh is locale dependent
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13995'>FLINK-13995</a>] - Fix shading of the licence information of netty
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13995">FLINK-13995</a>] - Fix shading of the licence information of netty
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13999'>FLINK-13999</a>] - Correct the documentation of MATCH_RECOGNIZE
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13999">FLINK-13999</a>] - Correct the documentation of MATCH_RECOGNIZE
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14009'>FLINK-14009</a>] - Cron jobs broken due to verifying incorrect NOTICE-binary file
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14009">FLINK-14009</a>] - Cron jobs broken due to verifying incorrect NOTICE-binary file
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14010'>FLINK-14010</a>] - Dispatcher &amp; JobManagers don&#39;t give up leadership when AM is shut down
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14010">FLINK-14010</a>] - Dispatcher &amp; JobManagers don&#39;t give up leadership when AM is shut down
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14043'>FLINK-14043</a>] - SavepointMigrationTestBase is super slow
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14043">FLINK-14043</a>] - SavepointMigrationTestBase is super slow
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14107'>FLINK-14107</a>] - Kinesis consumer record emitter deadlock under event time alignment
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14107">FLINK-14107</a>] - Kinesis consumer record emitter deadlock under event time alignment
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14175'>FLINK-14175</a>] - Upgrade KPL version in flink-connector-kinesis to fix application OOM
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14175">FLINK-14175</a>] - Upgrade KPL version in flink-connector-kinesis to fix application OOM
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14235'>FLINK-14235</a>] - Kafka010ProducerITCase&gt;KafkaProducerTestBase.testOneToOneAtLeastOnceCustomOperator fails on travis
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14235">FLINK-14235</a>] - Kafka010ProducerITCase&gt;KafkaProducerTestBase.testOneToOneAtLeastOnceCustomOperator fails on travis
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14315'>FLINK-14315</a>] - NPE with JobMaster.disconnectTaskManager
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14315">FLINK-14315</a>] - NPE with JobMaster.disconnectTaskManager
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14337'>FLINK-14337</a>] - HistoryServerTest.testHistoryServerIntegration failed on Travis
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14337">FLINK-14337</a>] - HistoryServerTest.testHistoryServerIntegration failed on Travis
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14347'>FLINK-14347</a>] - YARNSessionFIFOITCase.checkForProhibitedLogContents found a log with prohibited string
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14347">FLINK-14347</a>] - YARNSessionFIFOITCase.checkForProhibitedLogContents found a log with prohibited string
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14370'>FLINK-14370</a>] - KafkaProducerAtLeastOnceITCase&gt;KafkaProducerTestBase.testOneToOneAtLeastOnceRegularSink fails on Travis
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14370">FLINK-14370</a>] - KafkaProducerAtLeastOnceITCase&gt;KafkaProducerTestBase.testOneToOneAtLeastOnceRegularSink fails on Travis
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14398'>FLINK-14398</a>] - Further split input unboxing code into separate methods
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14398">FLINK-14398</a>] - Further split input unboxing code into separate methods
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14413'>FLINK-14413</a>] - shade-plugin ApacheNoticeResourceTransformer uses platform-dependent encoding
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14413">FLINK-14413</a>] - shade-plugin ApacheNoticeResourceTransformer uses platform-dependent encoding
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14434'>FLINK-14434</a>] - Dispatcher#createJobManagerRunner should not start JobManagerRunner
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14434">FLINK-14434</a>] - Dispatcher#createJobManagerRunner should not start JobManagerRunner
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14562'>FLINK-14562</a>] - RMQSource leaves idle consumer after closing
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14562">FLINK-14562</a>] - RMQSource leaves idle consumer after closing
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14589'>FLINK-14589</a>] - Redundant slot requests with the same AllocationID leads to inconsistent slot table
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14589">FLINK-14589</a>] - Redundant slot requests with the same AllocationID leads to inconsistent slot table
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-15036'>FLINK-15036</a>] - Container startup error will be handled out side of the YarnResourceManager&#39;s main thread
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15036">FLINK-15036</a>] - Container startup error will be handled out side of the YarnResourceManager&#39;s main thread
</li>
</ul>
-
+
<h2> Improvement
</h2>
<ul>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-12848'>FLINK-12848</a>] - Method equals() in RowTypeInfo should consider fieldsNames
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12848">FLINK-12848</a>] - Method equals() in RowTypeInfo should consider fieldsNames
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13729'>FLINK-13729</a>] - Update website generation dependencies
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13729">FLINK-13729</a>] - Update website generation dependencies
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13965'>FLINK-13965</a>] - Keep hasDeprecatedKeys and deprecatedKeys methods in ConfigOption and mark it with @Deprecated annotation
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13965">FLINK-13965</a>] - Keep hasDeprecatedKeys and deprecatedKeys methods in ConfigOption and mark it with @Deprecated annotation
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13967'>FLINK-13967</a>] - Generate full binary licensing via collect_license_files.sh
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13967">FLINK-13967</a>] - Generate full binary licensing via collect_license_files.sh
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13968'>FLINK-13968</a>] - Add travis check for the correctness of the binary licensing
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13968">FLINK-13968</a>] - Add travis check for the correctness of the binary licensing
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13991'>FLINK-13991</a>] - Add git exclusion for 1.9+ features to 1.8
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13991">FLINK-13991</a>] - Add git exclusion for 1.9+ features to 1.8
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14008'>FLINK-14008</a>] - Auto-generate binary licensing
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14008">FLINK-14008</a>] - Auto-generate binary licensing
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14104'>FLINK-14104</a>] - Bump Jackson to 2.10.1
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14104">FLINK-14104</a>] - Bump Jackson to 2.10.1
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14123'>FLINK-14123</a>] - Lower the default value of taskmanager.memory.fraction
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14123">FLINK-14123</a>] - Lower the default value of taskmanager.memory.fraction
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14215'>FLINK-14215</a>] - Add Docs for TM and JM Environment Variable Setting
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14215">FLINK-14215</a>] - Add Docs for TM and JM Environment Variable Setting
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14334'>FLINK-14334</a>] - ElasticSearch docs refer to non-existent ExceptionUtils.containsThrowable
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14334">FLINK-14334</a>] - ElasticSearch docs refer to non-existent ExceptionUtils.containsThrowable
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14639'>FLINK-14639</a>] - Fix the document of Metrics that has an error for `User Scope`
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14639">FLINK-14639</a>] - Fix the document of Metrics that has an error for `User Scope`
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14646'>FLINK-14646</a>] - Check non-null for key in KeyGroupStreamPartitioner
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14646">FLINK-14646</a>] - Check non-null for key in KeyGroupStreamPartitioner
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14995'>FLINK-14995</a>] - Kinesis NOTICE is incorrect
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14995">FLINK-14995</a>] - Kinesis NOTICE is incorrect
</li>
</ul>
</description>
-<pubDate>Wed, 11 Dec 2019 12:00:00 +0000</pubDate>
+<pubDate>Wed, 11 Dec 2019 13:00:00 +0100</pubDate>
<link>https://flink.apache.org/news/2019/12/11/release-1.8.3.html</link>
<guid isPermaLink="true">/news/2019/12/11/release-1.8.3.html</guid>
</item>
<item>
<title>Running Apache Flink on Kubernetes with KUDO</title>
-<description>A common use case for Apache Flink is streaming data analytics together with Apache Kafka, which provides a pub/sub model and durability for data streams. To achieve elastic scalability, both are typically deployed in clustered environments, and increasingly on top of container orchestration platforms like Kubernetes. The [Operator pattern](https://kubernetes.io/docs/concepts/extend-kubernetes/operator/) provides an extension mechanism to Kubernetes that captures human opera [...]
+<description><p>A common use case for Apache Flink is streaming data analytics together with Apache Kafka, which provides a pub/sub model and durability for data streams. To achieve elastic scalability, both are typically deployed in clustered environments, and increasingly on top of container orchestration platforms like Kubernetes. The <a href="https://kubernetes.io/docs/concepts/extend-kubernetes/operator/">Operator pattern</a> provides an extension mechani [...]
-In this blog post we demonstrate how to orchestrate a streaming data analytics application based on Flink and Kafka with KUDO. It consists of a Flink job that checks financial transactions for fraud, and two microservices that generate and display the transactions. You can find more details about this demo in the [KUDO Operators repository](https://github.com/kudobuilder/operators/tree/master/repository/flink/docs/demo/financial-fraud), including instructions for installing the dependencies.
+<p>In this blog post we demonstrate how to orchestrate a streaming data analytics application based on Flink and Kafka with KUDO. It consists of a Flink job that checks financial transactions for fraud, and two microservices that generate and display the transactions. You can find more details about this demo in the <a href="https://github.com/kudobuilder/operators/tree/master/repository/flink/docs/demo/financial-fraud">KUDO Operators repository</a>, including [...]
<p style="display: block; text-align: center; margin-top: 20px; margin-bottom: 20px">
- <img src="{{ site.baseurl }}/img/blog/2019-11-06-flink-kubernetes-kudo/flink-kudo-architecture.png" width="600px" alt="Application: My App"/>
+ <img src="/img/blog/2019-11-06-flink-kubernetes-kudo/flink-kudo-architecture.png" width="600px" alt="Application: My App" />
</p>
-## Prerequisites
+<h2 id="prerequisites">Prerequisites</h2>
-You can run this demo on your local machine using [minikube](https://github.com/kubernetes/minikube). The instructions below were tested with minikube v1.5.1 and Kubernetes v1.16.2 but should work on any Kubernetes version above v1.15.0. First, start a minikube cluster with enough capacity:
+<p>You can run this demo on your local machine using <a href="https://github.com/kubernetes/minikube">minikube</a>. The instructions below were tested with minikube v1.5.1 and Kubernetes v1.16.2 but should work on any Kubernetes version above v1.15.0. First, start a minikube cluster with enough capacity:</p>
-`minikube start --cpus=6 --memory=9216 --disk-size=10g`
+<p><code>minikube start --cpus=6 --memory=9216 --disk-size=10g</code></p>
-If you’re using a different way to provision Kubernetes, make sure you have at least 6 CPU Cores, 9 GB of RAM and 10 GB of disk space available.
+<p>If you’re using a different way to provision Kubernetes, make sure you have at least 6 CPU Cores, 9 GB of RAM and 10 GB of disk space available.</p>
-Install the `kubectl` CLI tool. The KUDO CLI is a plugin for the Kubernetes CLI. The official instructions for installing and setting up kubectl are [here](https://kubernetes.io/docs/tasks/tools/install-kubectl/).
+<p>Install the <code>kubectl</code> CLI tool. The KUDO CLI is a plugin for the Kubernetes CLI. The official instructions for installing and setting up kubectl are <a href="https://kubernetes.io/docs/tasks/tools/install-kubectl/">here</a>.</p>
-Next, let’s install the KUDO CLI. At the time of this writing, the latest KUDO version is v0.10.0. You can find the CLI binaries for download [here](https://github.com/kudobuilder/kudo/releases). Download the `kubectl-kudo` binary for your OS and architecture.
+<p>Next, let’s install the KUDO CLI. At the time of this writing, the latest KUDO version is v0.10.0. You can find the CLI binaries for download <a href="https://github.com/kudobuilder/kudo/releases">here</a>. Download the <code>kubectl-kudo</code> binary for your OS and architecture.</p>
-If you’re using Homebrew on MacOS, you can install the CLI via:
+<p>If you’re using Homebrew on MacOS, you can install the CLI via:</p>
-```
-$ brew tap kudobuilder/tap
+<div class="highlight"><pre><code>$ brew tap kudobuilder/tap
$ brew install kudo-cli
-```
+</code></pre></div>
-Now, let’s initialize KUDO on our Kubernetes cluster:
+<p>Now, let’s initialize KUDO on our Kubernetes cluster:</p>
-```
-$ kubectl kudo init
+<div class="highlight"><pre><code>$ kubectl kudo init
$KUDO_HOME has been configured at /Users/gerred/.kudo
-```
+</code></pre></div>
-This will create several resources. First, it will create the [Custom Resource Definitions](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/), [service account](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/), and [role bindings](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) necessary for KUDO to operate. It will also create an instance of the [KUDO controller](https://kudo.dev/docs/architecture.ht [...]
+<p>This will create several resources. First, it will create the <a href="https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/">Custom Resource Definitions</a>, <a href="https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/">service account</a>, and <a href="https://kubernetes.io/docs/reference/access-authn-authz/rbac/">role bindings</a> necessary for KUD [...]
-The KUDO CLI leverages the kubectl plugin system, which gives you all its functionality under `kubectl kudo`. This is a convenient way to install and deal with your KUDO Operators. For our demo, we use Kafka and Flink which depend on ZooKeeper. To make the ZooKeeper Operator available on the cluster, run:
+<p>The KUDO CLI leverages the kubectl plugin system, which gives you all its functionality under <code>kubectl kudo</code>. This is a convenient way to install and deal with your KUDO Operators. For our demo, we use Kafka and Flink which depend on ZooKeeper. To make the ZooKeeper Operator available on the cluster, run:</p>
-```
-$ kubectl kudo install zookeeper --version=0.3.0 --skip-instance
-```
+<div class="highlight"><pre><code>$ kubectl kudo install zookeeper --version=0.3.0 --skip-instance
+</code></pre></div>
-The --skip-instance flag skips the creation of a ZooKeeper instance. The flink-demo Operator that we’re going to install below will create it as a dependency instead. Now let’s make the Kafka and Flink Operators available the same way:
+<p>The –skip-instance flag skips the creation of a ZooKeeper instance. The flink-demo Operator that we’re going to install below will create it as a dependency instead. Now let’s make the Kafka and Flink Operators available the same way:</p>
-```
-$ kubectl kudo install kafka --version=1.2.0 --skip-instance
-```
+<div class="highlight"><pre><code>$ kubectl kudo install kafka --version=1.2.0 --skip-instance
+</code></pre></div>
-```
-$ kubectl kudo install flink --version=0.2.1 --skip-instance
-```
+<div class="highlight"><pre><code>$ kubectl kudo install flink --version=0.2.1 --skip-instance
+</code></pre></div>
-This installs all the Operator versions needed for our demo.
+<p>This installs all the Operator versions needed for our demo.</p>
-## Financial Fraud Demo
+<h2 id="financial-fraud-demo">Financial Fraud Demo</h2>
-In our financial fraud demo we have two microservices, called “generator” and “actor”. The generator produces transactions with random amounts and writes them into a Kafka topic. Occasionally, the value will be over 10,000 which is considered fraud for the purpose of this demo. The Flink job subscribes to the Kafka topic and detects fraudulent transactions. When it does, it submits them to another Kafka topic which the actor consumes. The actor simply displays each fraudulent transaction.
+<p>In our financial fraud demo we have two microservices, called “generator” and “actor”. The generator produces transactions with random amounts and writes them into a Kafka topic. Occasionally, the value will be over 10,000 which is considered fraud for the purpose of this demo. The Flink job subscribes to the Kafka topic and detects fraudulent transactions. When it does, it submits them to another Kafka topic which the actor consumes. The actor simply displays each fraudulent tr [...]
-The KUDO CLI by default installs Operators from the [official repository](https://github.com/kudobuilder/operators/), but it also supports installation from your local filesystem. This is useful if you want to develop your own Operator, or modify this demo for your own purposes.
+<p>The KUDO CLI by default installs Operators from the <a href="https://github.com/kudobuilder/operators/">official repository</a>, but it also supports installation from your local filesystem. This is useful if you want to develop your own Operator, or modify this demo for your own purposes.</p>
-First, clone the “kudobuilder/operators” repository via:
+<p>First, clone the “kudobuilder/operators” repository via:</p>
-```
-$ git clone https://github.com/kudobuilder/operators.git
-```
+<div class="highlight"><pre><code>$ git clone https://github.com/kudobuilder/operators.git
+</code></pre></div>
-Next, change into the “operators” directory and install the demo-operator from your local filesystem:
+<p>Next, change into the “operators” directory and install the demo-operator from your local filesystem:</p>
-```
-$ cd operators
+<div class="highlight"><pre><code>$ cd operators
$ kubectl kudo install repository/flink/docs/demo/financial-fraud/demo-operator --instance flink-demo
instance.kudo.dev/v1beta1/flink-demo created
-```
+</code></pre></div>
-This time we didn’t include the --skip-instance flag, so KUDO will actually deploy all the components, including Flink, Kafka, and ZooKeeper. KUDO orchestrates deployments and other lifecycle operations using [plans](https://kudo.dev/docs/concepts.html#plan) that were defined by the Operator developer. Plans are similar to [runbooks](https://en.wikipedia.org/wiki/Runbook) and encapsulate all the procedures required to operate the software. We can track the status of the deployment using [...]
+<p>This time we didn’t include the –skip-instance flag, so KUDO will actually deploy all the components, including Flink, Kafka, and ZooKeeper. KUDO orchestrates deployments and other lifecycle operations using <a href="https://kudo.dev/docs/concepts.html#plan">plans</a> that were defined by the Operator developer. Plans are similar to <a href="https://en.wikipedia.org/wiki/Runbook">runbooks</a> and encapsulate all the procedures required [...]
-```
-$ kubectl kudo plan status --instance flink-demo
+<div class="highlight"><pre><code>$ kubectl kudo plan status --instance flink-demo
Plan(s) for "flink-demo" in namespace "default":
.
└── flink-demo (Operator-Version: "flink-demo-0.1.4" Active-Plan: "deploy")
@@ -2024,12 +2023,11 @@ Plan(s) for "flink-demo" in namespace "default":
│ └── Step act (PENDING)
└── Phase flink-job [PENDING]
└── Step submit (PENDING)
-```
+</code></pre></div>
-The output shows that the “deploy” plan is in progress and that it consists of 4 phases: “dependencies”, “flink-cluster”, “demo” and “flink-job”. The “dependencies” phase includes steps for “zookeeper” and “kafka”. This is where both dependencies get installed, before KUDO continues to install the Flink cluster and the demo itself. We also see that ZooKeeper installation completed, and that Kafka installation is currently in progress. We can view details about Kafka’s deployment plan via:
+<p>The output shows that the “deploy” plan is in progress and that it consists of 4 phases: “dependencies”, “flink-cluster”, “demo” and “flink-job”. The “dependencies” phase includes steps for “zookeeper” and “kafka”. This is where both dependencies get installed, before KUDO continues to install the Flink cluster and the demo itself. We also see that ZooKeeper installation completed, and that Kafka installation is currently in progress. We can view details about Kafka’s deployment [...]
-```
-$ kubectl kudo plan status --instance flink-demo-kafka
+<div class="highlight"><pre><code>$ kubectl kudo plan status --instance flink-demo-kafka
Plan(s) for "flink-demo-kafka" in namespace "default":
.
└── flink-demo-kafka (Operator-Version: "kafka-1.2.0" Active-Plan: "deploy")
@@ -2040,28 +2038,26 @@ Plan(s) for "flink-demo-kafka" in namespace "default":
└── Phase not-allowed (serial strategy) [NOT ACTIVE]
└── Step not-allowed (serial strategy) [NOT ACTIVE]
└── not-allowed [NOT ACTIVE]
-```
+</code></pre></div>
-After Kafka was successfully installed the next phase “flink-cluster” will start and bring up, you guessed it, your flink-cluster. After this is done, the demo phase creates the generator and actor pods that generate and display transactions for this demo. Lastly, we have the flink-job phase in which we submit the actual FinancialFraudJob to the Flink cluster. Once the flink job is submitted, we will be able to see fraud logs in our actor pod shortly after.
+<p>After Kafka was successfully installed the next phase “flink-cluster” will start and bring up, you guessed it, your flink-cluster. After this is done, the demo phase creates the generator and actor pods that generate and display transactions for this demo. Lastly, we have the flink-job phase in which we submit the actual FinancialFraudJob to the Flink cluster. Once the flink job is submitted, we will be able to see fraud logs in our actor pod shortly after.</p>
-After a while, the state of all plans, phases and steps will change to “COMPLETE”. Now we can view the Flink dashboard to verify that our job is running. To access it from outside the Kubernetes cluster, first start the client proxy, then open the URL below in your browser:
+<p>After a while, the state of all plans, phases and steps will change to “COMPLETE”. Now we can view the Flink dashboard to verify that our job is running. To access it from outside the Kubernetes cluster, first start the client proxy, then open the URL below in your browser:</p>
-```
-$ kubectl proxy
-```
+<div class="highlight"><pre><code>$ kubectl proxy
+</code></pre></div>
-[http://127.0.0.1:8001/api/v1/namespaces/default/services/flink-demo-flink-jobmanager:ui/proxy/#/overview](http://127.0.0.1:8001/api/v1/namespaces/default/services/flink-demo-flink-jobmanager:ui/proxy/#/overview)
+<p><a href="http://127.0.0.1:8001/api/v1/namespaces/default/services/flink-demo-flink-jobmanager:ui/proxy/#/overview">http://127.0.0.1:8001/api/v1/namespaces/default/services/flink-demo-flink-jobmanager:ui/proxy/#/overview</a></p>
-It should look similar to this, depending on your local machine and how many cores you have available:
+<p>It should look similar to this, depending on your local machine and how many cores you have available:</p>
<p style="display: block; text-align: center; margin-top: 20px; margin-bottom: 20px">
- <img src="{{ site.baseurl }}/img/blog/2019-11-06-flink-kubernetes-kudo/flink-dashboard-ui.png" width="600px" alt="Application: My App"/>
+ <img src="/img/blog/2019-11-06-flink-kubernetes-kudo/flink-dashboard-ui.png" width="600px" alt="Application: My App" />
</p>
-The job is up and running and we should now be able to see fraudulent transaction in the logs of the actor pod:
+<p>The job is up and running and we should now be able to see fraudulent transaction in the logs of the actor pod:</p>
-```
-$ kubectl logs $(kubectl get pod -l actor=flink-demo -o jsonpath="{.items[0].metadata.name}")
+<div class="highlight"><pre><code>$ kubectl logs $(kubectl get pod -l actor=flink-demo -o jsonpath="{.items[0].metadata.name}")
Broker: flink-demo-kafka-kafka-0.flink-demo-kafka-svc:9093
Topic: fraud
@@ -2070,901 +2066,960 @@ Transaction{timestamp=1563395778000, origin=1, target='3', amount=8341}
Transaction{timestamp=1563395813000, origin=1, target='3', amount=8592}
Transaction{timestamp=1563395817000, origin=1, target='3', amount=2802}
Transaction{timestamp=1563395831000, origin=1, target='3', amount=160}}
-```
+</code></pre></div>
-If you add the “-f” flag to the previous command, you can follow along as more transactions are streaming in and are evaluated by our Flink job.
+<p>If you add the “-f” flag to the previous command, you can follow along as more transactions are streaming in and are evaluated by our Flink job.</p>
-## Conclusion
+<h2 id="conclusion">Conclusion</h2>
-In this blog post we demonstrated how to easily deploy an end-to-end streaming data application on Kubernetes using KUDO. We deployed a Flink job and two microservices, as well as all the required infrastructure - Flink, Kafka, and ZooKeeper using just a few kubectl commands. To find out more about KUDO, visit the [project website](https://kudo.dev) or join the community on [Slack](https://kubernetes.slack.com/messages/kudo/).
+<p>In this blog post we demonstrated how to easily deploy an end-to-end streaming data application on Kubernetes using KUDO. We deployed a Flink job and two microservices, as well as all the required infrastructure - Flink, Kafka, and ZooKeeper using just a few kubectl commands. To find out more about KUDO, visit the <a href="https://kudo.dev">project website</a> or join the community on <a href="https://kubernetes.slack.com/messages/kudo/">Sla [...]
</description>
-<pubDate>Mon, 09 Dec 2019 12:00:00 +0000</pubDate>
+<pubDate>Mon, 09 Dec 2019 13:00:00 +0100</pubDate>
<link>https://flink.apache.org/news/2019/12/09/flink-kubernetes-kudo.html</link>
<guid isPermaLink="true">/news/2019/12/09/flink-kubernetes-kudo.html</guid>
</item>
<item>
<title>How to query Pulsar Streams using Apache Flink</title>
-<description>In a previous [story](https://flink.apache.org/2019/05/03/pulsar-flink.html) on the Flink blog, we explained the different ways that [Apache Flink](https://flink.apache.org/) and [Apache Pulsar](https://pulsar.apache.org/) can integrate to provide elastic data processing at large scale. This blog post discusses the new developments and integrations between the two frameworks and showcases how you can leverage Pulsar’s built-in schema to query Pulsar streams in real time usi [...]
+<description><p>In a previous <a href="https://flink.apache.org/2019/05/03/pulsar-flink.html">story</a> on the Flink blog, we explained the different ways that <a href="https://flink.apache.org/">Apache Flink</a> and <a href="https://pulsar.apache.org/">Apache Pulsar</a> can integrate to provide elastic data processing at large scale. This blog post discusses the new developments and integrations between the two fra [...]
+<h1 id="a-short-intro-to-apache-pulsar">A short intro to Apache Pulsar</h1>
-# A short intro to Apache Pulsar
+<p>Apache Pulsar is a flexible pub/sub messaging system, backed by durable log storage. Some of the framework’s highlights include multi-tenancy, a unified message model, structured event streams and a cloud-native architecture that make it a perfect fit for a wide set of use cases, ranging from billing, payments and trading services all the way to the unification of the different messaging architectures in an organization. If you are interested in finding out more about Pulsar, yo [...]
-Apache Pulsar is a flexible pub/sub messaging system, backed by durable log storage. Some of the framework’s highlights include multi-tenancy, a unified message model, structured event streams and a cloud-native architecture that make it a perfect fit for a wide set of use cases, ranging from billing, payments and trading services all the way to the unification of the different messaging architectures in an organization. If you are interested in finding out more about Pulsar, you can vis [...]
+<h1 id="existing-pulsar--flink-integration-apache-flink-16">Existing Pulsar &amp; Flink integration (Apache Flink 1.6+)</h1>
+<p>The existing integration between Pulsar and Flink exploits Pulsar as a message queue in a Flink application. Flink developers can utilize Pulsar as a streaming source and streaming sink for their Flink applications by selecting a specific Pulsar source and connecting to their desired Pulsar cluster and topic:</p>
-# Existing Pulsar & Flink integration (Apache Flink 1.6+)
+<div class="highlight"><pre><code class="language-java"><span class="c1">// create and configure Pulsar consumer</span>
+<span class="n">PulsarSourceBuilder</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span><span class="n">builder</span> <span class="o">=</span> <span class="n">PulsarSourceBuilder</span>
+ <span class="o">.</span><span class="na">builder</span><span class="o">(</span><span class="k">new</span> <span class="nf">SimpleStringSchema</span><span class="o">())</span>
+ <span class="o">.</span><span class="na">serviceUrl</span><span class="o">(</span><span class="n">serviceUrl</span><span class="o">)</span>
+ <span class="o">.</span><span class="na">topic</span><span class="o">(</span><span class="n">inputTopic</span><span class="o">)</span>
+ <span class="o">.</span><span class="na">subsciptionName</span><span class="o">(</span><span class="n">subscription</span><span class="o">);</span>
+<span class="n">SourceFunction</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">src</span> <span class="o">=</span> <span class="n">builder</span><span class="o">.</span><span class="na">build</span><span class="o"&g [...]
+<span class="c1">// ingest DataStream with Pulsar consumer</span>
+<span class="n">DataStream</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">words</span> <span class="o">=</span> <span class="n">env</span><span class="o">.</span><span class="na">addSource</span><span class="o"> [...]
-The existing integration between Pulsar and Flink exploits Pulsar as a message queue in a Flink application. Flink developers can utilize Pulsar as a streaming source and streaming sink for their Flink applications by selecting a specific Pulsar source and connecting to their desired Pulsar cluster and topic:
+<p>Pulsar streams can then get connected to the Flink processing logic…</p>
-```java
-// create and configure Pulsar consumer
-PulsarSourceBuilder<String>builder = PulsarSourceBuilder
- .builder(new SimpleStringSchema())
- .serviceUrl(serviceUrl)
- .topic(inputTopic)
- .subsciptionName(subscription);
-SourceFunction<String> src = builder.build();
-// ingest DataStream with Pulsar consumer
-DataStream<String> words = env.addSource(src);
-```
-
-Pulsar streams can then get connected to the Flink processing logic…
-
-```java
-// perform computation on DataStream (here a simple WordCount)
-DataStream<WordWithCount> wc = words
- .flatmap((FlatMapFunction<String, WordWithCount>) (word, collector) -> {
- collector.collect(new WordWithCount(word, 1));
- })
+<div class="highlight"><pre><code class="language-java"><span class="c1">// perform computation on DataStream (here a simple WordCount)</span>
+<span class="n">DataStream</span><span class="o">&lt;</span><span class="n">WordWithCount</span><span class="o">&gt;</span> <span class="n">wc</span> <span class="o">=</span> <span class="n">words</span>
+ <span class="o">.</span><span class="na">flatmap</span><span class="o">((</span><span class="n">FlatMapFunction</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">WordWithCount</span><span class="o">&gt;)</span> <span class=" [...]
+ <span class="n">collector</span><span class="o">.</span><span class="na">collect</span><span class="o">(</span><span class="k">new</span> <span class="nf">WordWithCount</span><span class="o">(</span><span class="n">word</span><span class="o">,</span> <span class="mi">1</ [...]
+ <span class="o">})</span>
- .returns(WordWithCount.class)
- .keyBy("word")
- .timeWindow(Time.seconds(5))
- .reduce((ReduceFunction<WordWithCount>) (c1, c2) ->
- new WordWithCount(c1.word, c1.count + c2.count));
-```
-
-...and then get emitted back to Pulsar (used now as a sink), sending one’s computation results downstream, back to a Pulsar topic:
-
+ <span class="o">.</span><span class="na">returns</span><span class="o">(</span><span class="n">WordWithCount</span><span class="o">.</span><span class="na">class</span><span class="o">)</span>
+ <span class="o">.</span><span class="na">keyBy</span><span class="o">(</span><span class="s">&quot;word&quot;</span><span class="o">)</span>
+ <span class="o">.</span><span class="na">timeWindow</span><span class="o">(</span><span class="n">Time</span><span class="o">.</span><span class="na">seconds</span><span class="o">(</span><span class="mi">5</span><span class="o">))</span>
+ <span class="o">.</span><span class="na">reduce</span><span class="o">((</span><span class="n">ReduceFunction</span><span class="o">&lt;</span><span class="n">WordWithCount</span><span class="o">&gt;)</span> <span class="o">(</span><span class="n">c1</span><span class="o" [...]
+ <span class="k">new</span> <span class="nf">WordWithCount</span><span class="o">(</span><span class="n">c1</span><span class="o">.</span><span class="na">word</span><span class="o">,</span> <span class="n">c1</span><span class="o">.</span><span class="na">count</span> [...]
-```java
-// emit result via Pulsar producer
-wc.addSink(new FlinkPulsarProducer<>(
- serviceUrl,
- outputTopic,
- new AuthentificationDisabled(),
- wordWithCount -> wordWithCount.toString().getBytes(UTF_8),
- wordWithCount -> wordWithCount.word)
-);
-```
+<p>…and then get emitted back to Pulsar (used now as a sink), sending one’s computation results downstream, back to a Pulsar topic:</p>
-Although this is a great first integration step, the existing design is not leveraging the full power of Pulsar. Some shortcomings of the integration with Flink 1.6.0 relate to Pulsar neither being utilized as durable storage nor having schema integration with Flink, resulting in manual input when describing an application’s schema registry.
+<div class="highlight"><pre><code class="language-java"><span class="c1">// emit result via Pulsar producer </span>
+<span class="n">wc</span><span class="o">.</span><span class="na">addSink</span><span class="o">(</span><span class="k">new</span> <span class="n">FlinkPulsarProducer</span><span class="o">&lt;&gt;(</span>
+ <span class="n">serviceUrl</span><span class="o">,</span>
+ <span class="n">outputTopic</span><span class="o">,</span>
+ <span class="k">new</span> <span class="nf">AuthentificationDisabled</span><span class="o">(),</span>
+ <span class="n">wordWithCount</span> <span class="o">-&gt;</span> <span class="n">wordWithCount</span><span class="o">.</span><span class="na">toString</span><span class="o">().</span><span class="na">getBytes</span><span class="o">(</span><span class="n">UTF_8</span><span class=" [...]
+ <span class="n">wordWithCount</span> <span class="o">-&gt;</span> <span class="n">wordWithCount</span><span class="o">.</span><span class="na">word</span><span class="o">)</span>
+<span class="o">);</span></code></pre></div>
+<p>Although this is a great first integration step, the existing design is not leveraging the full power of Pulsar. Some shortcomings of the integration with Flink 1.6.0 relate to Pulsar neither being utilized as durable storage nor having schema integration with Flink, resulting in manual input when describing an application’s schema registry.</p>
-# Pulsar’s integration with Flink 1.9: Using Pulsar as a Flink catalog
+<h1 id="pulsars-integration-with-flink-19-using-pulsar-as-a-flink-catalog">Pulsar’s integration with Flink 1.9: Using Pulsar as a Flink catalog</h1>
-The latest integration between [Flink 1.9.0](https://flink.apache.org/downloads.html#apache-flink-191) and Pulsar addresses most of the previously mentioned shortcomings. The [contribution of Alibaba’s Blink to the Flink repository](https://flink.apache.org/news/2019/02/13/unified-batch-streaming-blink.html) adds many enhancements and new features to the processing framework that make the integration with Pulsar significantly more powerful and impactful. Flink 1.9.0 brings Pulsar schema [...]
+<p>The latest integration between <a href="https://flink.apache.org/downloads.html#apache-flink-191">Flink 1.9.0</a> and Pulsar addresses most of the previously mentioned shortcomings. The <a href="https://flink.apache.org/news/2019/02/13/unified-batch-streaming-blink.html">contribution of Alibaba’s Blink to the Flink repository</a> adds many enhancements and new features to the processing framework that make the integration with Pulsar s [...]
+<h1 id="leveraging-the-flink--pulsar-schema-integration">Leveraging the Flink &lt;&gt; Pulsar Schema Integration</h1>
-# Leveraging the Flink <> Pulsar Schema Integration
+<p>Before delving into the integration details and how you can use Pulsar schema with Flink, let us describe how schema in Pulsar works. Schema in Apache Pulsar already co-exists and serves as the representation of the data on the broker side of the framework, something that makes schema registry with external systems obsolete. Additionally, the data schema in Pulsar is associated with each topic so both producers and consumers send data with predefined schema information, while th [...]
-Before delving into the integration details and how you can use Pulsar schema with Flink, let us describe how schema in Pulsar works. Schema in Apache Pulsar already co-exists and serves as the representation of the data on the broker side of the framework, something that makes schema registry with external systems obsolete. Additionally, the data schema in Pulsar is associated with each topic so both producers and consumers send data with predefined schema information, while the broker [...]
-
-Below you can find an example of Pulsar’s schema on both the producer and consumer side. On the producer side, you can specify which schema you want to use and Pulsar then sends a POJO class without the need to perform any serialization/deserialization. Similarly, on the consumer end, you can also specify the data schema and upon receiving the data, Pulsar will automatically validate the schema information, fetch the schema of the given version and then deserialize the data back to a POJ [...]
+<p>Below you can find an example of Pulsar’s schema on both the producer and consumer side. On the producer side, you can specify which schema you want to use and Pulsar then sends a POJO class without the need to perform any serialization/deserialization. Similarly, on the consumer end, you can also specify the data schema and upon receiving the data, Pulsar will automatically validate the schema information, fetch the schema of the given version and then deserialize the data back [...]
-```java
-// Create producer with Struct schema and send messages
-Producer<User> producer = client.newProducer(Schema.AVRO(User.class)).create();
-producer.newMessage()
- .value(User.builder()
- .userName(“pulsar-user”)
- .userId(1L)
- .build())
- .send();
-```
+<div class="highlight"><pre><code class="language-java"><span class="c1">// Create producer with Struct schema and send messages</span>
+<span class="n">Producer</span><span class="o">&lt;</span><span class="n">User</span><span class="o">&gt;</span> <span class="n">producer</span> <span class="o">=</span> <span class="n">client</span><span class="o">.</span><span class="na">newProducer</span><span class="o" [...]
+<span class="n">producer</span><span class="o">.</span><span class="na">newMessage</span><span class="o">()</span>
+ <span class="o">.</span><span class="na">value</span><span class="o">(</span><span class="n">User</span><span class="o">.</span><span class="na">builder</span><span class="o">()</span>
+ <span class="o">.</span><span class="na">userName</span><span class="o">(</span><span class="err">“</span><span class="n">pulsar</span><span class="o">-</span><span class="n">user</span><span class="err">”</span><span class="o">)</span>
+ <span class="o">.</span><span class="na">userId</span><span class="o">(</span><span class="mi">1L</span><span class="o">)</span>
+ <span class="o">.</span><span class="na">build</span><span class="o">())</span>
+ <span class="o">.</span><span class="na">send</span><span class="o">();</span></code></pre></div>
-```java
-// Create consumer with Struct schema and receive messages
-Consumer<User> consumer = client.newCOnsumer(Schema.AVRO(User.class)).create();
-consumer.receive();
-```
+<div class="highlight"><pre><code class="language-java"><span class="c1">// Create consumer with Struct schema and receive messages</span>
+<span class="n">Consumer</span><span class="o">&lt;</span><span class="n">User</span><span class="o">&gt;</span> <span class="n">consumer</span> <span class="o">=</span> <span class="n">client</span><span class="o">.</span><span class="na">newCOnsumer</span><span class="o" [...]
+<span class="n">consumer</span><span class="o">.</span><span class="na">receive</span><span class="o">();</span></code></pre></div>
-Let’s assume we have an application that specifies a schema to the producer and/or consumer. Upon receiving the schema information, the producer (or consumer) — that is connected to the broker — will transfer such information so that the broker can then perform schema registration, validations and schema compatibility checks before returning or rejecting the schema as illustrated in the diagram below:
+<p>Let’s assume we have an application that specifies a schema to the producer and/or consumer. Upon receiving the schema information, the producer (or consumer) — that is connected to the broker — will transfer such information so that the broker can then perform schema registration, validations and schema compatibility checks before returning or rejecting the schema as illustrated in the diagram below:</p>
<center>
-<img src="{{ site.baseurl }}/img/blog/flink-pulsar-sql-blog-post-visual.png" width="600px" alt="Pulsar Schema"/>
+<img src="/img/blog/flink-pulsar-sql-blog-post-visual.png" width="600px" alt="Pulsar Schema" />
</center>
-<br>
+<p><br /></p>
-Not only is Pulsar able to handle and store the schema information, but is additionally able to handle any schema evolution — where necessary. Pulsar will effectively manage any schema evolution in the broker, keeping track of all different versions of your schema while performing any necessary compatibility checks.
-
-Moreover, when messages are published on the producer side, Pulsar will tag each message with the schema version as part of each message’s metadata. On the consumer side, when the message is received and the metadata is deserialized, Pulsar will check the schema version associated with this message and will fetch the corresponding schema information from the broker. As a result, when Pulsar integrates with a Flink application it uses the pre-existing schema information and maps individua [...]
-
-For the cases when Flink users do not interact with schema directly or make use of primitive schema (for example, using a topic to store a string or long number), Pulsar will either convert the message payload into a Flink row, called ‘value’ or — for the cases of structured schema types, like JSON and AVRO — Pulsar will extract the individual fields from the schema information and will map the fields to Flink’s type system. Finally, all metadata information associated with each message [...]
+<p>Not only is Pulsar able to handle and store the schema information, but is additionally able to handle any schema evolution — where necessary. Pulsar will effectively manage any schema evolution in the broker, keeping track of all different versions of your schema while performing any necessary compatibility checks.</p>
+
+<p>Moreover, when messages are published on the producer side, Pulsar will tag each message with the schema version as part of each message’s metadata. On the consumer side, when the message is received and the metadata is deserialized, Pulsar will check the schema version associated with this message and will fetch the corresponding schema information from the broker. As a result, when Pulsar integrates with a Flink application it uses the pre-existing schema information and maps [...]
+<p>For the cases when Flink users do not interact with schema directly or make use of primitive schema (for example, using a topic to store a string or long number), Pulsar will either convert the message payload into a Flink row, called ‘value’ or — for the cases of structured schema types, like JSON and AVRO — Pulsar will extract the individual fields from the schema information and will map the fields to Flink’s type system. Finally, all metadata information associated with eac [...]
<center>
-<img src="{{ site.baseurl }}/img/blog/flink-pulsar-sql-blog-post-visual-primitive-avro-schema.png" width="600px" alt="Primitive and AVRO Schema"/>
+<img src="/img/blog/flink-pulsar-sql-blog-post-visual-primitive-avro-schema.png" width="600px" alt="Primitive and AVRO Schema" />
</center>
-<br>
+<p><br /></p>
-Once all the schema information is mapped to Flink’s type system, you can start building a Pulsar source, sink or catalog in Flink based on the specified schema information as illustrated below:
+<p>Once all the schema information is mapped to Flink’s type system, you can start building a Pulsar source, sink or catalog in Flink based on the specified schema information as illustrated below:</p>
-# Flink & Pulsar: Read data from Pulsar
+<h1 id="flink--pulsar-read-data-from-pulsar">Flink &amp; Pulsar: Read data from Pulsar</h1>
-* Create a Pulsar source for streaming queries
+<ul>
+ <li>Create a Pulsar source for streaming queries</li>
+</ul>
-```java
-val env = StreamExecutionEnvironment.getExecutionEnvironment
-val props = new Properties()
-props.setProperty("service.url", "pulsar://...")
-props.setProperty("admin.url", "http://...")
-props.setProperty("partitionDiscoveryIntervalMillis", "5000")
-props.setProperty("startingOffsets", "earliest")
-props.setProperty("topic", "test-source-topic")
-val source = new FlinkPulsarSource(props)
-// you don't need to provide a type information to addSource since FlinkPulsarSource is ResultTypeQueryable
-val dataStream = env.addSource(source)(null)
+<div class="highlight"><pre><code class="language-java"><span class="n">val</span> <span class="n">env</span> <span class="o">=</span> <span class="n">StreamExecutionEnvironment</span><span class="o">.</span><span class="na">getExecutionEnvironment</span>
+<span class="n">val</span> <span class="n">props</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">Properties</span><span class="o">()</span>
+<span class="n">props</span><span class="o">.</span><span class="na">setProperty</span><span class="o">(</span><span class="s">&quot;service.url&quot;</span><span class="o">,</span> <span class="s">&quot;pulsar://...&quot;</span><span class="o">)</span>
+<span class="n">props</span><span class="o">.</span><span class="na">setProperty</span><span class="o">(</span><span class="s">&quot;admin.url&quot;</span><span class="o">,</span> <span class="s">&quot;http://...&quot;</span><span class="o">)</span>
+<span class="n">props</span><span class="o">.</span><span class="na">setProperty</span><span class="o">(</span><span class="s">&quot;partitionDiscoveryIntervalMillis&quot;</span><span class="o">,</span> <span class="s">&quot;5000&quot;</span><span class="o">)</span>
+<span class="n">props</span><span class="o">.</span><span class="na">setProperty</span><span class="o">(</span><span class="s">&quot;startingOffsets&quot;</span><span class="o">,</span> <span class="s">&quot;earliest&quot;</span><span class="o">)</span>
+<span class="n">props</span><span class="o">.</span><span class="na">setProperty</span><span class="o">(</span><span class="s">&quot;topic&quot;</span><span class="o">,</span> <span class="s">&quot;test-source-topic&quot;</span><span class="o">)</span>
+<span class="n">val</span> <span class="n">source</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">FlinkPulsarSource</span><span class="o">(</span><span class="n">props</span><span class="o">)</span>
+<span class="c1">// you don&#39;t need to provide a type information to addSource since FlinkPulsarSource is ResultTypeQueryable</span>
+<span class="n">val</span> <span class="n">dataStream</span> <span class="o">=</span> <span class="n">env</span><span class="o">.</span><span class="na">addSource</span><span class="o">(</span><span class="n">source</span><span class="o">)(</span><span class="kc">null</span& [...]
-// chain operations on dataStream of Row and sink the output
-// end method chaining
+<span class="c1">// chain operations on dataStream of Row and sink the output</span>
+<span class="c1">// end method chaining</span>
-env.execute()
-```
+<span class="n">env</span><span class="o">.</span><span class="na">execute</span><span class="o">()</span></code></pre></div>
+
+<ul>
+ <li>Register topics in Pulsar as streaming tables</li>
+</ul>
-* Register topics in Pulsar as streaming tables
+<div class="highlight"><pre><code class="language-java"><span class="n">val</span> <span class="n">env</span> <span class="o">=</span> <span class="n">StreamExecutionEnvironment</span><span class="o">.</span><span class="na">getExecutionEnvironment</span>
+<span class="n">val</span> <span class="n">tEnv</span> <span class="o">=</span> <span class="n">StreamTableEnvironment</span><span class="o">.</span><span class="na">create</span><span class="o">(</span><span class="n">env</span><span class="o">)</span>
-```java
-val env = StreamExecutionEnvironment.getExecutionEnvironment
-val tEnv = StreamTableEnvironment.create(env)
+<span class="n">val</span> <span class="n">prop</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">Properties</span><span class="o">()</span>
+<span class="n">prop</span><span class="o">.</span><span class="na">setProperty</span><span class="o">(</span><span class="s">&quot;service.url&quot;</span><span class="o">,</span> <span class="n">serviceUrl</span><span class="o">)</span>
+<span class="n">prop</span><span class="o">.</span><span class="na">setProperty</span><span class="o">(</span><span class="s">&quot;admin.url&quot;</span><span class="o">,</span> <span class="n">adminUrl</span><span class="o">)</span>
+<span class="n">prop</span><span class="o">.</span><span class="na">setProperty</span><span class="o">(</span><span class="s">&quot;flushOnCheckpoint&quot;</span><span class="o">,</span> <span class="s">&quot;true&quot;</span><span class="o">)</span>
+<span class="n">prop</span><span class="o">.</span><span class="na">setProperty</span><span class="o">(</span><span class="s">&quot;failOnWrite&quot;</span><span class="o">,</span> <span class="s">&quot;true&quot;</span><span class="o">)</span>
+<span class="n">props</span><span class="o">.</span><span class="na">setProperty</span><span class="o">(</span><span class="s">&quot;topic&quot;</span><span class="o">,</span> <span class="s">&quot;test-sink-topic&quot;</span><span class="o">)</span>
-val prop = new Properties()
-prop.setProperty("service.url", serviceUrl)
-prop.setProperty("admin.url", adminUrl)
-prop.setProperty("flushOnCheckpoint", "true")
-prop.setProperty("failOnWrite", "true")
-props.setProperty("topic", "test-sink-topic")
+<span class="n">tEnv</span>
+ <span class="o">.</span><span class="na">connect</span><span class="o">(</span><span class="k">new</span> <span class="nf">Pulsar</span><span class="o">().</span><span class="na">properties</span><span class="o">(</span><span class="n">props</span><span class="o">))</span>
+ <span class="o">.</span><span class="na">inAppendMode</span><span class="o">()</span>
+ <span class="o">.</span><span class="na">registerTableSource</span><span class="o">(</span><span class="s">&quot;sink-table&quot;</span><span class="o">)</span>
-tEnv
- .connect(new Pulsar().properties(props))
- .inAppendMode()
- .registerTableSource("sink-table")
+<span class="n">val</span> <span class="n">sql</span> <span class="o">=</span> <span class="s">&quot;INSERT INTO sink-table .....&quot;</span>
+<span class="n">tEnv</span><span class="o">.</span><span class="na">sqlUpdate</span><span class="o">(</span><span class="n">sql</span><span class="o">)</span>
+<span class="n">env</span><span class="o">.</span><span class="na">execute</span><span class="o">()</span></code></pre></div>
-val sql = "INSERT INTO sink-table ....."
-tEnv.sqlUpdate(sql)
-env.execute()
-```
+<h1 id="flink--pulsar-write-data-to-pulsar">Flink &amp; Pulsar: Write data to Pulsar</h1>
-# Flink & Pulsar: Write data to Pulsar
+<ul>
+ <li>Create a Pulsar sink for streaming queries</li>
+</ul>
-* Create a Pulsar sink for streaming queries
+<div class="highlight"><pre><code class="language-java"><span class="n">val</span> <span class="n">env</span> <span class="o">=</span> <span class="n">StreamExecutionEnvironment</span><span class="o">.</span><span class="na">getExecutionEnvironment</span>
+<span class="n">val</span> <span class="n">stream</span> <span class="o">=</span> <span class="o">.....</span>
-```java
-val env = StreamExecutionEnvironment.getExecutionEnvironment
-val stream = .....
+<span class="n">val</span> <span class="n">prop</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">Properties</span><span class="o">()</span>
+<span class="n">prop</span><span class="o">.</span><span class="na">setProperty</span><span class="o">(</span><span class="s">&quot;service.url&quot;</span><span class="o">,</span> <span class="n">serviceUrl</span><span class="o">)</span>
+<span class="n">prop</span><span class="o">.</span><span class="na">setProperty</span><span class="o">(</span><span class="s">&quot;admin.url&quot;</span><span class="o">,</span> <span class="n">adminUrl</span><span class="o">)</span>
+<span class="n">prop</span><span class="o">.</span><span class="na">setProperty</span><span class="o">(</span><span class="s">&quot;flushOnCheckpoint&quot;</span><span class="o">,</span> <span class="s">&quot;true&quot;</span><span class="o">)</span>
+<span class="n">prop</span><span class="o">.</span><span class="na">setProperty</span><span class="o">(</span><span class="s">&quot;failOnWrite&quot;</span><span class="o">,</span> <span class="s">&quot;true&quot;</span><span class="o">)</span>
+<span class="n">props</span><span class="o">.</span><span class="na">setProperty</span><span class="o">(</span><span class="s">&quot;topic&quot;</span><span class="o">,</span> <span class="s">&quot;test-sink-topic&quot;</span><span class="o">)</span>
-val prop = new Properties()
-prop.setProperty("service.url", serviceUrl)
-prop.setProperty("admin.url", adminUrl)
-prop.setProperty("flushOnCheckpoint", "true")
-prop.setProperty("failOnWrite", "true")
-props.setProperty("topic", "test-sink-topic")
+<span class="n">stream</span><span class="o">.</span><span class="na">addSink</span><span class="o">(</span><span class="k">new</span> <span class="nf">FlinkPulsarSink</span><span class="o">(</span><span class="n">prop</span><span class="o">,</span> <span class="n">DummyTopicKe [...]
+<span class="n">env</span><span class="o">.</span><span class="na">execute</span><span class="o">()</span></code></pre></div>
-stream.addSink(new FlinkPulsarSink(prop, DummyTopicKeyExtractor))
-env.execute()
-```
+<ul>
+ <li>Write a streaming table to Pulsar</li>
+</ul>
-* Write a streaming table to Pulsar
+<div class="highlight"><pre><code class="language-java"><span class="n">val</span> <span class="n">env</span> <span class="o">=</span> <span class="n">StreamExecutionEnvironment</span><span class="o">.</span><span class="na">getExecutionEnvironment</span>
+<span class="n">val</span> <span class="n">tEnv</span> <span class="o">=</span> <span class="n">StreamTableEnvironment</span><span class="o">.</span><span class="na">create</span><span class="o">(</span><span class="n">env</span><span class="o">)</span>
-```java
-val env = StreamExecutionEnvironment.getExecutionEnvironment
-val tEnv = StreamTableEnvironment.create(env)
+<span class="n">val</span> <span class="n">prop</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">Properties</span><span class="o">()</span>
+<span class="n">prop</span><span class="o">.</span><span class="na">setProperty</span><span class="o">(</span><span class="s">&quot;service.url&quot;</span><span class="o">,</span> <span class="n">serviceUrl</span><span class="o">)</span>
+<span class="n">prop</span><span class="o">.</span><span class="na">setProperty</span><span class="o">(</span><span class="s">&quot;admin.url&quot;</span><span class="o">,</span> <span class="n">adminUrl</span><span class="o">)</span>
+<span class="n">prop</span><span class="o">.</span><span class="na">setProperty</span><span class="o">(</span><span class="s">&quot;flushOnCheckpoint&quot;</span><span class="o">,</span> <span class="s">&quot;true&quot;</span><span class="o">)</span>
+<span class="n">prop</span><span class="o">.</span><span class="na">setProperty</span><span class="o">(</span><span class="s">&quot;failOnWrite&quot;</span><span class="o">,</span> <span class="s">&quot;true&quot;</span><span class="o">)</span>
+<span class="n">props</span><span class="o">.</span><span class="na">setProperty</span><span class="o">(</span><span class="s">&quot;topic&quot;</span><span class="o">,</span> <span class="s">&quot;test-sink-topic&quot;</span><span class="o">)</span>
-val prop = new Properties()
-prop.setProperty("service.url", serviceUrl)
-prop.setProperty("admin.url", adminUrl)
-prop.setProperty("flushOnCheckpoint", "true")
-prop.setProperty("failOnWrite", "true")
-props.setProperty("topic", "test-sink-topic")
+<span class="n">tEnv</span>
+ <span class="o">.</span><span class="na">connect</span><span class="o">(</span><span class="k">new</span> <span class="nf">Pulsar</span><span class="o">().</span><span class="na">properties</span><span class="o">(</span><span class="n">props</span><span class="o">))</span>
+ <span class="o">.</span><span class="na">inAppendMode</span><span class="o">()</span>
+ <span class="o">.</span><span class="na">registerTableSource</span><span class="o">(</span><span class="s">&quot;sink-table&quot;</span><span class="o">)</span>
-tEnv
- .connect(new Pulsar().properties(props))
- .inAppendMode()
- .registerTableSource("sink-table")
+<span class="n">val</span> <span class="n">sql</span> <span class="o">=</span> <span class="s">&quot;INSERT INTO sink-table .....&quot;</span>
+<span class="n">tEnv</span><span class="o">.</span><span class="na">sqlUpdate</span><span class="o">(</span><span class="n">sql</span><span class="o">)</span>
+<span class="n">env</span><span class="o">.</span><span class="na">execute</span><span class="o">()</span></code></pre></div>
-val sql = "INSERT INTO sink-table ....."
-tEnv.sqlUpdate(sql)
-env.execute()
-```
+<p>In every instance, Flink developers only need to specify the properties of how Flink will connect to a Pulsar cluster without worrying about any schema registry, or serialization/deserialization actions and register the Pulsar cluster as a source, sink or streaming table in Flink. Once all three elements are put together, Pulsar can then be registered as a catalog in Flink, something that drastically simplifies how you process and query data like, for example, writing a program [...]
-In every instance, Flink developers only need to specify the properties of how Flink will connect to a Pulsar cluster without worrying about any schema registry, or serialization/deserialization actions and register the Pulsar cluster as a source, sink or streaming table in Flink. Once all three elements are put together, Pulsar can then be registered as a catalog in Flink, something that drastically simplifies how you process and query data like, for example, writing a program to query [...]
+<h1 id="next-steps--future-integration">Next Steps &amp; Future Integration</h1>
+<p>The goal of the integration between Pulsar and Flink is to simplify how developers use the two frameworks to build a unified data processing stack. As we progress from the classical Lamda architectures — where an online, speeding layer is combined with an offline, batch layer to run data computations — Flink and Pulsar present a great combination in providing a truly unified data processing stack. We see Flink as a unified computation engine, handling both online (streaming) and [...]
-# Next Steps & Future Integration
+<p>There is still a lot of ongoing work and effort from both communities in getting the integration even better, such as a new source API (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface">FLIP-27</a>) that will allow the <a href="http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Discussion-Flink-Pulsar-Connector-td22019.html">contribution of the Pulsar connectors to the Flink communit [...]
-The goal of the integration between Pulsar and Flink is to simplify how developers use the two frameworks to build a unified data processing stack. As we progress from the classical Lamda architectures — where an online, speeding layer is combined with an offline, batch layer to run data computations — Flink and Pulsar present a great combination in providing a truly unified data processing stack. We see Flink as a unified computation engine, handling both online (streaming) and offline [...]
-
-There is still a lot of ongoing work and effort from both communities in getting the integration even better, such as a new source API ([FLIP-27](https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface)) that will allow the [contribution of the Pulsar connectors to the Flink community](http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Discussion-Flink-Pulsar-Connector-td22019.html) as well as a new subscription type called `Key_Shared` subscrip [...]
-
-You can find a more detailed overview of the integration work between the two communities in this [recording video](https://youtu.be/3sBXXfgl5vs) from Flink Forward Europe 2019 or sign up to the [Flink dev mailing list](https://flink.apache.org/community.html#mailing-lists) for the latest contribution and integration efforts between Flink and Pulsar.
+<p>You can find a more detailed overview of the integration work between the two communities in this <a href="https://youtu.be/3sBXXfgl5vs">recording video</a> from Flink Forward Europe 2019 or sign up to the <a href="https://flink.apache.org/community.html#mailing-lists">Flink dev mailing list</a> for the latest contribution and integration efforts between Flink and Pulsar.</p>
</description>
-<pubDate>Mon, 25 Nov 2019 12:00:00 +0000</pubDate>
+<pubDate>Mon, 25 Nov 2019 13:00:00 +0100</pubDate>
<link>https://flink.apache.org/news/2019/11/25/query-pulsar-streams-using-apache-flink.html</link>
<guid isPermaLink="true">/news/2019/11/25/query-pulsar-streams-using-apache-flink.html</guid>
</item>
<item>
<title>Apache Flink 1.9.1 Released</title>
-<description>The Apache Flink community released the first bugfix version of the Apache Flink 1.9 series.
+<description><p>The Apache Flink community released the first bugfix version of the Apache Flink 1.9 series.</p>
-This release includes 96 fixes and minor improvements for Flink 1.9.0. The list below includes a detailed list of all fixes and improvements.
+<p>This release includes 96 fixes and minor improvements for Flink 1.9.0. The list below includes a detailed list of all fixes and improvements.</p>
-We highly recommend all users to upgrade to Flink 1.9.1.
+<p>We highly recommend all users to upgrade to Flink 1.9.1.</p>
-Updated Maven dependencies:
+<p>Updated Maven dependencies:</p>
-```xml
-<dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-java</artifactId>
- <version>1.9.1</version>
-</dependency>
-<dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-streaming-java_2.11</artifactId>
- <version>1.9.1</version>
-</dependency>
-<dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-clients_2.11</artifactId>
- <version>1.9.1</version>
-</dependency>
-```
+<div class="highlight"><pre><code class="language-xml"><span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-java<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.9.1<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span>
+<span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-streaming-java_2.11<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.9.1<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span>
+<span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-clients_2.11<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.9.1<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span></code></pre></div>
-You can find the binaries on the updated [Downloads page]({{ site.baseurl }}/downloads.html).
+<p>You can find the binaries on the updated <a href="/downloads.html">Downloads page</a>.</p>
-List of resolved issues:
+<p>List of resolved issues:</p>
<h2> Bug
</h2>
<ul>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-11630'>FLINK-11630</a>] - TaskExecutor does not wait for Task termination when terminating itself
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11630">FLINK-11630</a>] - TaskExecutor does not wait for Task termination when terminating itself
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13490'>FLINK-13490</a>] - Fix if one column value is null when reading JDBC, the following values are all null
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13490">FLINK-13490</a>] - Fix if one column value is null when reading JDBC, the following values are all null
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13941'>FLINK-13941</a>] - Prevent data-loss by not cleaning up small part files from S3.
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13941">FLINK-13941</a>] - Prevent data-loss by not cleaning up small part files from S3.
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-12501'>FLINK-12501</a>] - AvroTypeSerializer does not work with types generated by avrohugger
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12501">FLINK-12501</a>] - AvroTypeSerializer does not work with types generated by avrohugger
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13386'>FLINK-13386</a>] - Fix some frictions in the new default Web UI
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13386">FLINK-13386</a>] - Fix some frictions in the new default Web UI
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13526'>FLINK-13526</a>] - Switching to a non existing catalog or database crashes sql-client
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13526">FLINK-13526</a>] - Switching to a non existing catalog or database crashes sql-client
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13568'>FLINK-13568</a>] - DDL create table doesn&#39;t allow STRING data type
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13568">FLINK-13568</a>] - DDL create table doesn&#39;t allow STRING data type
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13805'>FLINK-13805</a>] - Bad Error Message when TaskManager is lost
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13805">FLINK-13805</a>] - Bad Error Message when TaskManager is lost
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13806'>FLINK-13806</a>] - Metric Fetcher floods the JM log with errors when TM is lost
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13806">FLINK-13806</a>] - Metric Fetcher floods the JM log with errors when TM is lost
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14010'>FLINK-14010</a>] - Dispatcher &amp; JobManagers don&#39;t give up leadership when AM is shut down
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14010">FLINK-14010</a>] - Dispatcher &amp; JobManagers don&#39;t give up leadership when AM is shut down
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14145'>FLINK-14145</a>] - CompletedCheckpointStore#getLatestCheckpoint(true) returns wrong checkpoint
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14145">FLINK-14145</a>] - CompletedCheckpointStore#getLatestCheckpoint(true) returns wrong checkpoint
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13059'>FLINK-13059</a>] - Cassandra Connector leaks Semaphore on Exception and hangs on close
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13059">FLINK-13059</a>] - Cassandra Connector leaks Semaphore on Exception and hangs on close
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13534'>FLINK-13534</a>] - Unable to query Hive table with decimal column
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13534">FLINK-13534</a>] - Unable to query Hive table with decimal column
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13562'>FLINK-13562</a>] - Throws exception when FlinkRelMdColumnInterval meets two stage stream group aggregate
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13562">FLINK-13562</a>] - Throws exception when FlinkRelMdColumnInterval meets two stage stream group aggregate
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13563'>FLINK-13563</a>] - TumblingGroupWindow should implement toString method
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13563">FLINK-13563</a>] - TumblingGroupWindow should implement toString method
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13564'>FLINK-13564</a>] - Throw exception if constant with YEAR TO MONTH resolution was used for group windows
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13564">FLINK-13564</a>] - Throw exception if constant with YEAR TO MONTH resolution was used for group windows
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13588'>FLINK-13588</a>] - StreamTask.handleAsyncException throws away the exception cause
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13588">FLINK-13588</a>] - StreamTask.handleAsyncException throws away the exception cause
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13653'>FLINK-13653</a>] - ResultStore should avoid using RowTypeInfo when creating a result
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13653">FLINK-13653</a>] - ResultStore should avoid using RowTypeInfo when creating a result
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13711'>FLINK-13711</a>] - Hive array values not properly displayed in SQL CLI
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13711">FLINK-13711</a>] - Hive array values not properly displayed in SQL CLI
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13737'>FLINK-13737</a>] - flink-dist should add provided dependency on flink-examples-table
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13737">FLINK-13737</a>] - flink-dist should add provided dependency on flink-examples-table
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13738'>FLINK-13738</a>] - Fix NegativeArraySizeException in LongHybridHashTable
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13738">FLINK-13738</a>] - Fix NegativeArraySizeException in LongHybridHashTable
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13742'>FLINK-13742</a>] - Fix code generation when aggregation contains both distinct aggregate with and without filter
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13742">FLINK-13742</a>] - Fix code generation when aggregation contains both distinct aggregate with and without filter
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13760'>FLINK-13760</a>] - Fix hardcode Scala version dependency in hive connector
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13760">FLINK-13760</a>] - Fix hardcode Scala version dependency in hive connector
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13761'>FLINK-13761</a>] - `SplitStream` should be deprecated because `SplitJavaStream` is deprecated
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13761">FLINK-13761</a>] - `SplitStream` should be deprecated because `SplitJavaStream` is deprecated
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13789'>FLINK-13789</a>] - Transactional Id Generation fails due to user code impacting formatting string
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13789">FLINK-13789</a>] - Transactional Id Generation fails due to user code impacting formatting string
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13823'>FLINK-13823</a>] - Incorrect debug log in CompileUtils
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13823">FLINK-13823</a>] - Incorrect debug log in CompileUtils
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13825'>FLINK-13825</a>] - The original plugins dir is not restored after e2e test run
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13825">FLINK-13825</a>] - The original plugins dir is not restored after e2e test run
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13831'>FLINK-13831</a>] - Free Slots / All Slots display error
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13831">FLINK-13831</a>] - Free Slots / All Slots display error
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13887'>FLINK-13887</a>] - Ensure defaultInputDependencyConstraint to be non-null when setting it in ExecutionConfig
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13887">FLINK-13887</a>] - Ensure defaultInputDependencyConstraint to be non-null when setting it in ExecutionConfig
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13897'>FLINK-13897</a>] - OSS FS NOTICE file is placed in wrong directory
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13897">FLINK-13897</a>] - OSS FS NOTICE file is placed in wrong directory
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13933'>FLINK-13933</a>] - Hive Generic UDTF can not be used in table API both stream and batch mode
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13933">FLINK-13933</a>] - Hive Generic UDTF can not be used in table API both stream and batch mode
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13936'>FLINK-13936</a>] - NOTICE-binary is outdated
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13936">FLINK-13936</a>] - NOTICE-binary is outdated
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13966'>FLINK-13966</a>] - Jar sorting in collect_license_files.sh is locale dependent
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13966">FLINK-13966</a>] - Jar sorting in collect_license_files.sh is locale dependent
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14009'>FLINK-14009</a>] - Cron jobs broken due to verifying incorrect NOTICE-binary file
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14009">FLINK-14009</a>] - Cron jobs broken due to verifying incorrect NOTICE-binary file
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14049'>FLINK-14049</a>] - Update error message for failed partition updates to include task name
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14049">FLINK-14049</a>] - Update error message for failed partition updates to include task name
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14076'>FLINK-14076</a>] - &#39;ClassNotFoundException: KafkaException&#39; on Flink v1.9 w/ checkpointing
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14076">FLINK-14076</a>] - &#39;ClassNotFoundException: KafkaException&#39; on Flink v1.9 w/ checkpointing
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14107'>FLINK-14107</a>] - Kinesis consumer record emitter deadlock under event time alignment
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14107">FLINK-14107</a>] - Kinesis consumer record emitter deadlock under event time alignment
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14119'>FLINK-14119</a>] - Clean idle state for RetractableTopNFunction
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14119">FLINK-14119</a>] - Clean idle state for RetractableTopNFunction
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14139'>FLINK-14139</a>] - Fix potential memory leak of rest server when using session/standalone cluster
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14139">FLINK-14139</a>] - Fix potential memory leak of rest server when using session/standalone cluster
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14140'>FLINK-14140</a>] - The Flink Logo Displayed in Flink Python Shell is Broken
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14140">FLINK-14140</a>] - The Flink Logo Displayed in Flink Python Shell is Broken
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14150'>FLINK-14150</a>] - Unnecessary __pycache__ directories appears in pyflink.zip
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14150">FLINK-14150</a>] - Unnecessary __pycache__ directories appears in pyflink.zip
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14288'>FLINK-14288</a>] - Add Py4j NOTICE for source release
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14288">FLINK-14288</a>] - Add Py4j NOTICE for source release
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13892'>FLINK-13892</a>] - HistoryServerTest failed on Travis
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13892">FLINK-13892</a>] - HistoryServerTest failed on Travis
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14043'>FLINK-14043</a>] - SavepointMigrationTestBase is super slow
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14043">FLINK-14043</a>] - SavepointMigrationTestBase is super slow
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-12164'>FLINK-12164</a>] - JobMasterTest.testJobFailureWhenTaskExecutorHeartbeatTimeout is unstable
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12164">FLINK-12164</a>] - JobMasterTest.testJobFailureWhenTaskExecutorHeartbeatTimeout is unstable
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-9900'>FLINK-9900</a>] - Fix unstable test ZooKeeperHighAvailabilityITCase#testRestoreBehaviourWithFaultyStateHandles
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-9900">FLINK-9900</a>] - Fix unstable test ZooKeeperHighAvailabilityITCase#testRestoreBehaviourWithFaultyStateHandles
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13484'>FLINK-13484</a>] - ConnectedComponents end-to-end test instable with NoResourceAvailableException
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13484">FLINK-13484</a>] - ConnectedComponents end-to-end test instable with NoResourceAvailableException
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13489'>FLINK-13489</a>] - Heavy deployment end-to-end test fails on Travis with TM heartbeat timeout
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13489">FLINK-13489</a>] - Heavy deployment end-to-end test fails on Travis with TM heartbeat timeout
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13514'>FLINK-13514</a>] - StreamTaskTest.testAsyncCheckpointingConcurrentCloseAfterAcknowledge unstable
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13514">FLINK-13514</a>] - StreamTaskTest.testAsyncCheckpointingConcurrentCloseAfterAcknowledge unstable
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13530'>FLINK-13530</a>] - AbstractServerTest failed on Travis
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13530">FLINK-13530</a>] - AbstractServerTest failed on Travis
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13585'>FLINK-13585</a>] - Fix sporadical deallock in TaskAsyncCallTest#testSetsUserCodeClassLoader()
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13585">FLINK-13585</a>] - Fix sporadical deallock in TaskAsyncCallTest#testSetsUserCodeClassLoader()
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13599'>FLINK-13599</a>] - Kinesis end-to-end test failed on Travis
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13599">FLINK-13599</a>] - Kinesis end-to-end test failed on Travis
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13663'>FLINK-13663</a>] - SQL Client end-to-end test for modern Kafka failed on Travis
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13663">FLINK-13663</a>] - SQL Client end-to-end test for modern Kafka failed on Travis
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13688'>FLINK-13688</a>] - HiveCatalogUseBlinkITCase.testBlinkUdf constantly failed
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13688">FLINK-13688</a>] - HiveCatalogUseBlinkITCase.testBlinkUdf constantly failed
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13739'>FLINK-13739</a>] - BinaryRowTest.testWriteString() fails in some environments
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13739">FLINK-13739</a>] - BinaryRowTest.testWriteString() fails in some environments
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13746'>FLINK-13746</a>] - Elasticsearch (v2.3.5) sink end-to-end test fails on Travis
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13746">FLINK-13746</a>] - Elasticsearch (v2.3.5) sink end-to-end test fails on Travis
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13769'>FLINK-13769</a>] - BatchFineGrainedRecoveryITCase.testProgram failed on Travis
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13769">FLINK-13769</a>] - BatchFineGrainedRecoveryITCase.testProgram failed on Travis
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13807'>FLINK-13807</a>] - Flink-avro unit tests fails if the character encoding in the environment is not default to UTF-8
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13807">FLINK-13807</a>] - Flink-avro unit tests fails if the character encoding in the environment is not default to UTF-8
</li>
</ul>
-
<h2> Improvement
</h2>
<ul>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13965'>FLINK-13965</a>] - Keep hasDeprecatedKeys and deprecatedKeys methods in ConfigOption and mark it with @Deprecated annotation
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13965">FLINK-13965</a>] - Keep hasDeprecatedKeys and deprecatedKeys methods in ConfigOption and mark it with @Deprecated annotation
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-9941'>FLINK-9941</a>] - Flush in ScalaCsvOutputFormat before close method
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-9941">FLINK-9941</a>] - Flush in ScalaCsvOutputFormat before close method
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13336'>FLINK-13336</a>] - Remove the legacy batch fault tolerance page and redirect it to the new task failure recovery page
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13336">FLINK-13336</a>] - Remove the legacy batch fault tolerance page and redirect it to the new task failure recovery page
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13380'>FLINK-13380</a>] - Improve the usability of Flink session cluster on Kubernetes
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13380">FLINK-13380</a>] - Improve the usability of Flink session cluster on Kubernetes
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13819'>FLINK-13819</a>] - Introduce RpcEndpoint State
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13819">FLINK-13819</a>] - Introduce RpcEndpoint State
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13845'>FLINK-13845</a>] - Drop all the content of removed &quot;Checkpointed&quot; interface
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13845">FLINK-13845</a>] - Drop all the content of removed &quot;Checkpointed&quot; interface
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13957'>FLINK-13957</a>] - Log dynamic properties on job submission
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13957">FLINK-13957</a>] - Log dynamic properties on job submission
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13967'>FLINK-13967</a>] - Generate full binary licensing via collect_license_files.sh
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13967">FLINK-13967</a>] - Generate full binary licensing via collect_license_files.sh
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13968'>FLINK-13968</a>] - Add travis check for the correctness of the binary licensing
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13968">FLINK-13968</a>] - Add travis check for the correctness of the binary licensing
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13449'>FLINK-13449</a>] - Add ARM architecture to MemoryArchitecture
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13449">FLINK-13449</a>] - Add ARM architecture to MemoryArchitecture
</li>
</ul>
-
<h2> Documentation
</h2>
<ul>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13105'>FLINK-13105</a>] - Add documentation for blink planner&#39;s built-in functions
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13105">FLINK-13105</a>] - Add documentation for blink planner&#39;s built-in functions
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13277'>FLINK-13277</a>] - add documentation of Hive source/sink
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13277">FLINK-13277</a>] - add documentation of Hive source/sink
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13354'>FLINK-13354</a>] - Add documentation for how to use blink planner
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13354">FLINK-13354</a>] - Add documentation for how to use blink planner
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13355'>FLINK-13355</a>] - Add documentation for Temporal Table Join in blink planner
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13355">FLINK-13355</a>] - Add documentation for Temporal Table Join in blink planner
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13356'>FLINK-13356</a>] - Add documentation for TopN and Deduplication in blink planner
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13356">FLINK-13356</a>] - Add documentation for TopN and Deduplication in blink planner
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13359'>FLINK-13359</a>] - Add documentation for DDL introduction
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13359">FLINK-13359</a>] - Add documentation for DDL introduction
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13362'>FLINK-13362</a>] - Add documentation for Kafka &amp; ES &amp; FileSystem DDL
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13362">FLINK-13362</a>] - Add documentation for Kafka &amp; ES &amp; FileSystem DDL
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13363'>FLINK-13363</a>] - Add documentation for streaming aggregate performance tunning.
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13363">FLINK-13363</a>] - Add documentation for streaming aggregate performance tunning.
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13706'>FLINK-13706</a>] - add documentation of how to use Hive functions in Flink
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13706">FLINK-13706</a>] - add documentation of how to use Hive functions in Flink
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13942'>FLINK-13942</a>] - Add Overview page for Getting Started section
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13942">FLINK-13942</a>] - Add Overview page for Getting Started section
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13863'>FLINK-13863</a>] - Update Operations Playground to Flink 1.9.0
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13863">FLINK-13863</a>] - Update Operations Playground to Flink 1.9.0
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13937'>FLINK-13937</a>] - Fix wrong hive dependency version in documentation
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13937">FLINK-13937</a>] - Fix wrong hive dependency version in documentation
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13830'>FLINK-13830</a>] - The Document about Cluster on yarn have some problems
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13830">FLINK-13830</a>] - The Document about Cluster on yarn have some problems
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-14160'>FLINK-14160</a>] - Extend Operations Playground with --backpressure option
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14160">FLINK-14160</a>] - Extend Operations Playground with --backpressure option
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13388'>FLINK-13388</a>] - Update UI screenshots in the documentation to the new default Web Frontend
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13388">FLINK-13388</a>] - Update UI screenshots in the documentation to the new default Web Frontend
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13415'>FLINK-13415</a>] - Document how to use hive connector in scala shell
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13415">FLINK-13415</a>] - Document how to use hive connector in scala shell
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13517'>FLINK-13517</a>] - Restructure Hive Catalog documentation
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13517">FLINK-13517</a>] - Restructure Hive Catalog documentation
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13643'>FLINK-13643</a>] - Document the workaround for users with a different minor Hive version
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13643">FLINK-13643</a>] - Document the workaround for users with a different minor Hive version
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13757'>FLINK-13757</a>] - Fix wrong description of "IS NOT TRUE" function documentation
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13757">FLINK-13757</a>] - Fix wrong description of "IS NOT TRUE" function documentation
</li>
</ul>
</description>
-<pubDate>Fri, 18 Oct 2019 12:00:00 +0000</pubDate>
+<pubDate>Fri, 18 Oct 2019 14:00:00 +0200</pubDate>
<link>https://flink.apache.org/news/2019/10/18/release-1.9.1.html</link>
<guid isPermaLink="true">/news/2019/10/18/release-1.9.1.html</guid>
</item>
<item>
<title>The State Processor API: How to Read, write and modify the state of Flink applications</title>
-<description>Whether you are running Apache Flink<sup>Ⓡ</sup> in production or evaluated Flink as a computation framework in the past, you've probably found yourself asking the question: How can I access, write or update state in a Flink savepoint? Ask no more! [Apache Flink 1.9.0](https://flink.apache.org/news/2019/08/22/release-1.9.0.html) introduces the [State Processor API](https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/libs/state_processor_api.html), [...]
-
-In this post, we explain why this feature is a big step for Flink, what you can use it for, and how to use it. Finally, we will discuss the future of the State Processor API and how it aligns with our plans to evolve Flink into a system for [unified batch and stream processing](https://flink.apache.org/news/2019/02/13/unified-batch-streaming-blink.html).
+<description><p>Whether you are running Apache Flink<sup>Ⓡ</sup> in production or evaluated Flink as a computation framework in the past, you’ve probably found yourself asking the question: How can I access, write or update state in a Flink savepoint? Ask no more! <a href="https://flink.apache.org/news/2019/08/22/release-1.9.0.html">Apache Flink 1.9.0</a> introduces the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.9/de [...]
-## Stateful Stream Processing with Apache Flink until Flink 1.9
+<p>In this post, we explain why this feature is a big step for Flink, what you can use it for, and how to use it. Finally, we will discuss the future of the State Processor API and how it aligns with our plans to evolve Flink into a system for <a href="https://flink.apache.org/news/2019/02/13/unified-batch-streaming-blink.html">unified batch and stream processing</a>.</p>
-All non-trivial stream processing applications are stateful and most of them are designed to run for months or years. Over time, many of them accumulate a lot of valuable state that can be very expensive or even impossible to rebuild if it gets lost due to a failure. In order to guarantee the consistency and durability of application state, Flink featured a sophisticated checkpointing and recovery mechanism from very early on. With every release, the Flink community has added more and mo [...]
+<h2 id="stateful-stream-processing-with-apache-flink-until-flink-19">Stateful Stream Processing with Apache Flink until Flink 1.9</h2>
-However, a feature that was commonly requested by Flink users was the ability to access the state of an application “from the outside”. This request was motivated by the need to validate or debug the state of an application, to migrate the state of an application to another application, to evolve an application from the Heap State Backend to the RocksDB State Backend, or to import the initial state of an application from an external system like a relational database.
+<p>All non-trivial stream processing applications are stateful and most of them are designed to run for months or years. Over time, many of them accumulate a lot of valuable state that can be very expensive or even impossible to rebuild if it gets lost due to a failure. In order to guarantee the consistency and durability of application state, Flink featured a sophisticated checkpointing and recovery mechanism from very early on. With every release, the Flink community has added mo [...]
-Despite all those convincing reasons to expose application state externally, your access options have been fairly limited until now. Flink's Queryable State feature only supports key-lookups (point queries) and does not guarantee the consistency of returned values (the value of a key might be different before and after an application recovered from a failure). Moreover, queryable state cannot be used to add or modify the state of an application. Also, savepoints, which are consistent [...]
+<p>However, a feature that was commonly requested by Flink users was the ability to access the state of an application “from the outside”. This request was motivated by the need to validate or debug the state of an application, to migrate the state of an application to another application, to evolve an application from the Heap State Backend to the RocksDB State Backend, or to import the initial state of an application from an external system like a relational database.</p>
-## Reading and Writing Application State with the State Processor API
+<p>Despite all those convincing reasons to expose application state externally, your access options have been fairly limited until now. Flink’s Queryable State feature only supports key-lookups (point queries) and does not guarantee the consistency of returned values (the value of a key might be different before and after an application recovered from a failure). Moreover, queryable state cannot be used to add or modify the state of an application. Also, savepoints, which are consi [...]
-The State Processor API that comes with Flink 1.9 is a true game-changer in how you can work with application state! In a nutshell, it extends the DataSet API with Input and OutputFormats to read and write savepoint or checkpoint data. Due to the [interoperability of DataSet and Table API](https://ci.apache.org/projects/flink/flink-docs-master/dev/table/common.html#integration-with-datastream-and-dataset-api), you can even use relational Table API or SQL queries to analyze and process st [...]
+<h2 id="reading-and-writing-application-state-with-the-state-processor-api">Reading and Writing Application State with the State Processor API</h2>
-For example, you can take a savepoint of a running stream processing application and analyze it with a DataSet batch program to verify that the application behaves correctly. Or you can read a batch of data from any store, preprocess it, and write the result to a savepoint that you use to bootstrap the state of a streaming application. It's also possible to fix inconsistent state entries now. Finally, the State Processor API opens up many ways to evolve a stateful application that we [...]
+<p>The State Processor API that comes with Flink 1.9 is a true game-changer in how you can work with application state! In a nutshell, it extends the DataSet API with Input and OutputFormats to read and write savepoint or checkpoint data. Due to the <a href="https://ci.apache.org/projects/flink/flink-docs-master/dev/table/common.html#integration-with-datastream-and-dataset-api">interoperability of DataSet and Table API</a>, you can even use relational Table AP [...]
-## Mapping Application State to DataSets
+<p>For example, you can take a savepoint of a running stream processing application and analyze it with a DataSet batch program to verify that the application behaves correctly. Or you can read a batch of data from any store, preprocess it, and write the result to a savepoint that you use to bootstrap the state of a streaming application. It’s also possible to fix inconsistent state entries now. Finally, the State Processor API opens up many ways to evolve a stateful application th [...]
-The State Processor API maps the state of a streaming application to one or more data sets that can be separately processed. In order to be able to use the API, you need to understand how this mapping works.
-
-But let's first have a look at what a stateful Flink job looks like. A Flink job is composed of operators, typically one or more source operators, a few operators for the actual processing, and one or more sink operators. Each operator runs in parallel in one or more tasks and can work with different types of state. An operator can have zero, one, or more *“operator states”* which are organized as lists that are scoped to the operator's tasks. If the operator is applied on a keye [...]
-
-The following figure shows the application “MyApp” which consists of three operators called “Src”, “Proc”, and “Snk”. Src has one operator state (os1), Proc has one operator state (os2) and two keyed states (ks1, ks2) and Snk is stateless.
+<h2 id="mapping-application-state-to-datasets">Mapping Application State to DataSets</h2>
+
+<p>The State Processor API maps the state of a streaming application to one or more data sets that can be separately processed. In order to be able to use the API, you need to understand how this mapping works.</p>
+
+<p>But let’s first have a look at what a stateful Flink job looks like. A Flink job is composed of operators, typically one or more source operators, a few operators for the actual processing, and one or more sink operators. Each operator runs in parallel in one or more tasks and can work with different types of state. An operator can have zero, one, or more <em>“operator states”</em> which are organized as lists that are scoped to the operator’s tasks. If the operator [...]
+
+<p>The following figure shows the application “MyApp” which consists of three operators called “Src”, “Proc”, and “Snk”. Src has one operator state (os1), Proc has one operator state (os2) and two keyed states (ks1, ks2) and Snk is stateless.</p>
<p style="display: block; text-align: center; margin-top: 20px; margin-bottom: 20px">
- <img src="{{ site.baseurl }}/img/blog/2019-09-13-state-processor-api-blog/application-my-app-state-processor-api.png" width="600px" alt="Application: My App"/>
+ <img src="/img/blog/2019-09-13-state-processor-api-blog/application-my-app-state-processor-api.png" width="600px" alt="Application: My App" />
</p>
-A savepoint or checkpoint of MyApp consists of the data of all states, organized in a way that the states of each task can be restored. When processing the data of a savepoint (or checkpoint) with a batch job, we need a mental model that maps the data of the individual tasks' states into data sets or tables. In fact, we can think of a savepoint as a database. Every operator (identified by its UID) represents a namespace. Each operator state of an operator is mapped to a dedicated tab [...]
+<p>A savepoint or checkpoint of MyApp consists of the data of all states, organized in a way that the states of each task can be restored. When processing the data of a savepoint (or checkpoint) with a batch job, we need a mental model that maps the data of the individual tasks’ states into data sets or tables. In fact, we can think of a savepoint as a database. Every operator (identified by its UID) represents a namespace. Each operator state of an operator is mapped to a dedicate [...]
<p style="display: block; text-align: center; margin-top: 20px; margin-bottom: 20px">
- <img src="{{ site.baseurl }}/img/blog/2019-09-13-state-processor-api-blog/database-my-app-state-processor-api.png" width="600px" alt="Database: My App"/>
+ <img src="/img/blog/2019-09-13-state-processor-api-blog/database-my-app-state-processor-api.png" width="600px" alt="Database: My App" />
</p>
-The figure shows how the values of Src's operator state are mapped to a table with one column and five rows, one row for all list entries across all parallel tasks of Src. Operator state os2 of the operator “Proc” is similarly mapped to an individual table. The keyed states ks1 and ks2 are combined to a single table with three columns, one for the key, one for ks1 and one for ks2. The keyed table holds one row for each distinct key of both keyed states. Since the operator “Snk” does [...]
+<p>The figure shows how the values of Src’s operator state are mapped to a table with one column and five rows, one row for all list entries across all parallel tasks of Src. Operator state os2 of the operator “Proc” is similarly mapped to an individual table. The keyed states ks1 and ks2 are combined to a single table with three columns, one for the key, one for ks1 and one for ks2. The keyed table holds one row for each distinct key of both keyed states. Since the operator “Snk” [...]
-The State Processor API now offers methods to create, load, and write a savepoint. You can read a DataSet from a loaded savepoint or convert a DataSet into a state and add it to a savepoint. DataSets can be processed with the full feature set of the DataSet API. With these building blocks, all of the before-mentioned use cases (and more) can be addressed. Please have a look at the [documentation](https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/libs/state_processor_api.htm [...]
+<p>The State Processor API now offers methods to create, load, and write a savepoint. You can read a DataSet from a loaded savepoint or convert a DataSet into a state and add it to a savepoint. DataSets can be processed with the full feature set of the DataSet API. With these building blocks, all of the before-mentioned use cases (and more) can be addressed. Please have a look at the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/libs/state_process [...]
-## Why DataSet API?
+<h2 id="why-dataset-api">Why DataSet API?</h2>
-In case you are familiar with [Flink's roadmap](https://flink.apache.org/roadmap.html), you might be surprised that the State Processor API is based on the DataSet API. The Flink community plans to extend the DataStream API with the concept of *BoundedStreams* and deprecate the DataSet API. When designing this feature, we also evaluated the DataStream API or Table API but neither could provide the right feature set yet. Since we didn't want to block this feature on the progress o [...]
+<p>In case you are familiar with <a href="https://flink.apache.org/roadmap.html">Flink’s roadmap</a>, you might be surprised that the State Processor API is based on the DataSet API. The Flink community plans to extend the DataStream API with the concept of <em>BoundedStreams</em> and deprecate the DataSet API. When designing this feature, we also evaluated the DataStream API or Table API but neither could provide the right feature set yet. Since w [...]
-## Summary
+<h2 id="summary">Summary</h2>
-Flink users have requested a feature to access and modify the state of streaming applications from the outside for a long time. With the State Processor API, Flink 1.9.0 finally exposes application state as a data format that can be manipulated. This feature opens up many new possibilities for how users can maintain and manage Flink streaming applications, including arbitrary evolution of stream applications and exporting and bootstrapping of application state. To put it concisely, the S [...]
+<p>Flink users have requested a feature to access and modify the state of streaming applications from the outside for a long time. With the State Processor API, Flink 1.9.0 finally exposes application state as a data format that can be manipulated. This feature opens up many new possibilities for how users can maintain and manage Flink streaming applications, including arbitrary evolution of stream applications and exporting and bootstrapping of application state. To put it concise [...]
</description>
-<pubDate>Fri, 13 Sep 2019 12:00:00 +0000</pubDate>
+<pubDate>Fri, 13 Sep 2019 14:00:00 +0200</pubDate>
<link>https://flink.apache.org/feature/2019/09/13/state-processor-api.html</link>
<guid isPermaLink="true">/feature/2019/09/13/state-processor-api.html</guid>
</item>
<item>
<title>Apache Flink 1.8.2 Released</title>
-<description>The Apache Flink community released the second bugfix version of the Apache Flink 1.8 series.
+<description><p>The Apache Flink community released the second bugfix version of the Apache Flink 1.8 series.</p>
-This release includes 23 fixes and minor improvements for Flink 1.8.1. The list below includes a detailed list of all fixes and improvements.
+<p>This release includes 23 fixes and minor improvements for Flink 1.8.1. The list below includes a detailed list of all fixes and improvements.</p>
-We highly recommend all users to upgrade to Flink 1.8.2.
+<p>We highly recommend all users to upgrade to Flink 1.8.2.</p>
-Updated Maven dependencies:
+<p>Updated Maven dependencies:</p>
-```xml
-<dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-java</artifactId>
- <version>1.8.2</version>
-</dependency>
-<dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-streaming-java_2.11</artifactId>
- <version>1.8.2</version>
-</dependency>
-<dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-clients_2.11</artifactId>
- <version>1.8.2</version>
-</dependency>
-```
+<div class="highlight"><pre><code class="language-xml"><span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-java<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.8.2<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span>
+<span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-streaming-java_2.11<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.8.2<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span>
+<span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-clients_2.11<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.8.2<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span></code></pre></div>
-You can find the binaries on the updated [Downloads page]({{ site.baseurl }}/downloads.html).
+<p>You can find the binaries on the updated <a href="/downloads.html">Downloads page</a>.</p>
-List of resolved issues:
+<p>List of resolved issues:</p>
<h2> Bug
</h2>
<ul>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13941'>FLINK-13941</a>] - Prevent data-loss by not cleaning up small part files from S3.
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13941">FLINK-13941</a>] - Prevent data-loss by not cleaning up small part files from S3.
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-9526'>FLINK-9526</a>] - BucketingSink end-to-end test failed on Travis
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-9526">FLINK-9526</a>] - BucketingSink end-to-end test failed on Travis
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10368'>FLINK-10368</a>] - &#39;Kerberized YARN on Docker test&#39; unstable
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10368">FLINK-10368</a>] - &#39;Kerberized YARN on Docker test&#39; unstable
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-12319'>FLINK-12319</a>] - StackOverFlowError in cep.nfa.sharedbuffer.SharedBuffer
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12319">FLINK-12319</a>] - StackOverFlowError in cep.nfa.sharedbuffer.SharedBuffer
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-12736'>FLINK-12736</a>] - ResourceManager may release TM with allocated slots
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12736">FLINK-12736</a>] - ResourceManager may release TM with allocated slots
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-12889'>FLINK-12889</a>] - Job keeps in FAILING state
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12889">FLINK-12889</a>] - Job keeps in FAILING state
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13059'>FLINK-13059</a>] - Cassandra Connector leaks Semaphore on Exception; hangs on close
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13059">FLINK-13059</a>] - Cassandra Connector leaks Semaphore on Exception; hangs on close
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13159'>FLINK-13159</a>] - java.lang.ClassNotFoundException when restore job
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13159">FLINK-13159</a>] - java.lang.ClassNotFoundException when restore job
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13367'>FLINK-13367</a>] - Make ClosureCleaner detect writeReplace serialization override
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13367">FLINK-13367</a>] - Make ClosureCleaner detect writeReplace serialization override
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13369'>FLINK-13369</a>] - Recursive closure cleaner ends up with stackOverflow in case of circular dependency
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13369">FLINK-13369</a>] - Recursive closure cleaner ends up with stackOverflow in case of circular dependency
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13394'>FLINK-13394</a>] - Use fallback unsafe secure MapR in nightly.sh
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13394">FLINK-13394</a>] - Use fallback unsafe secure MapR in nightly.sh
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13484'>FLINK-13484</a>] - ConnectedComponents end-to-end test instable with NoResourceAvailableException
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13484">FLINK-13484</a>] - ConnectedComponents end-to-end test instable with NoResourceAvailableException
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13499'>FLINK-13499</a>] - Remove dependency on MapR artifact repository
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13499">FLINK-13499</a>] - Remove dependency on MapR artifact repository
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13508'>FLINK-13508</a>] - CommonTestUtils#waitUntilCondition() may attempt to sleep with negative time
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13508">FLINK-13508</a>] - CommonTestUtils#waitUntilCondition() may attempt to sleep with negative time
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13586'>FLINK-13586</a>] - Method ClosureCleaner.clean broke backward compatibility between 1.8.0 and 1.8.1
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13586">FLINK-13586</a>] - Method ClosureCleaner.clean broke backward compatibility between 1.8.0 and 1.8.1
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13761'>FLINK-13761</a>] - `SplitStream` should be deprecated because `SplitJavaStream` is deprecated
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13761">FLINK-13761</a>] - `SplitStream` should be deprecated because `SplitJavaStream` is deprecated
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13789'>FLINK-13789</a>] - Transactional Id Generation fails due to user code impacting formatting string
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13789">FLINK-13789</a>] - Transactional Id Generation fails due to user code impacting formatting string
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13806'>FLINK-13806</a>] - Metric Fetcher floods the JM log with errors when TM is lost
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13806">FLINK-13806</a>] - Metric Fetcher floods the JM log with errors when TM is lost
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13807'>FLINK-13807</a>] - Flink-avro unit tests fails if the character encoding in the environment is not default to UTF-8
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13807">FLINK-13807</a>] - Flink-avro unit tests fails if the character encoding in the environment is not default to UTF-8
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-13897'>FLINK-13897</a>] - OSS FS NOTICE file is placed in wrong directory
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13897">FLINK-13897</a>] - OSS FS NOTICE file is placed in wrong directory
</li>
</ul>
<h2> Improvement
</h2>
<ul>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-12578'>FLINK-12578</a>] - Use secure URLs for Maven repositories
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12578">FLINK-12578</a>] - Use secure URLs for Maven repositories
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-12741'>FLINK-12741</a>] - Update docs about Kafka producer fault tolerance guarantees
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12741">FLINK-12741</a>] - Update docs about Kafka producer fault tolerance guarantees
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-12749'>FLINK-12749</a>] - Add Flink Operations Playground documentation
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12749">FLINK-12749</a>] - Add Flink Operations Playground documentation
</li>
</ul>
</description>
-<pubDate>Wed, 11 Sep 2019 12:00:00 +0000</pubDate>
+<pubDate>Wed, 11 Sep 2019 14:00:00 +0200</pubDate>
<link>https://flink.apache.org/news/2019/09/11/release-1.8.2.html</link>
<guid isPermaLink="true">/news/2019/09/11/release-1.8.2.html</guid>
</item>
<item>
<title>Flink Community Update - September'19</title>
-<description>This has been an exciting, fast-paced year for the Apache Flink community. But with over 10k messages across the mailing lists, 3k Jira tickets and 2k pull requests, it is not easy to keep up with the latest state of the project. Plus everything happening around it. With that in mind, we want to bring back regular community updates to the Flink blog.
-
-The first post in the series takes you on an little detour across the year, to freshen up and make sure you're all up to date.
+<description><p>This has been an exciting, fast-paced year for the Apache Flink community. But with over 10k messages across the mailing lists, 3k Jira tickets and 2k pull requests, it is not easy to keep up with the latest state of the project. Plus everything happening around it. With that in mind, we want to bring back regular community updates to the Flink blog.</p>
+
+<p>The first post in the series takes you on an little detour across the year, to freshen up and make sure you’re all up to date.</p>
+
+<div class="page-toc">
+<ul id="markdown-toc">
+ <li><a href="#the-year-so-far-in-flink" id="markdown-toc-the-year-so-far-in-flink">The Year (so far) in Flink</a> <ul>
+ <li><a href="#integration-of-the-chinese-speaking-community" id="markdown-toc-integration-of-the-chinese-speaking-community">Integration of the Chinese-speaking community</a></li>
+ <li><a href="#improving-flinks-documentation" id="markdown-toc-improving-flinks-documentation">Improving Flink’s Documentation</a></li>
+ <li><a href="#adjusting-the-contribution-process-and-experience" id="markdown-toc-adjusting-the-contribution-process-and-experience">Adjusting the Contribution Process and Experience</a></li>
+ <li><a href="#new-committers-and-pmc-members" id="markdown-toc-new-committers-and-pmc-members">New Committers and PMC Members</a> <ul>
+ <li><a href="#new-pmc-members" id="markdown-toc-new-pmc-members">New PMC Members</a></li>
+ <li><a href="#new-committers" id="markdown-toc-new-committers">New Committers</a></li>
+ </ul>
+ </li>
+ </ul>
+ </li>
+ <li><a href="#the-bigger-picture" id="markdown-toc-the-bigger-picture">The Bigger Picture</a></li>
+ <li><a href="#upcoming-events" id="markdown-toc-upcoming-events">Upcoming Events</a> <ul>
+ <li><a href="#north-america" id="markdown-toc-north-america">North America</a></li>
+ <li><a href="#europe" id="markdown-toc-europe">Europe</a></li>
+ <li><a href="#asia" id="markdown-toc-asia">Asia</a></li>
+ </ul>
+ </li>
+</ul>
-{% toc %}
+</div>
-# The Year (so far) in Flink
+<h1 id="the-year-so-far-in-flink">The Year (so far) in Flink</h1>
-Two major versions were released this year: [Flink 1.8](https://flink.apache.org/news/2019/04/09/release-1.8.0.html) and [Flink 1.9](https://flink.apache.org/news/2019/08/22/release-1.9.0.html); paving the way for the goal of making Flink the first framework to seamlessly support stream and batch processing with a single, unified runtime. The [contribution of Blink](https://flink.apache.org/news/2019/02/13/unified-batch-streaming-blink.html) to Apache Flink was key in accelerating the pa [...]
+<p>Two major versions were released this year: <a href="https://flink.apache.org/news/2019/04/09/release-1.8.0.html">Flink 1.8</a> and <a href="https://flink.apache.org/news/2019/08/22/release-1.9.0.html">Flink 1.9</a>; paving the way for the goal of making Flink the first framework to seamlessly support stream and batch processing with a single, unified runtime. The <a href="https://flink.apache.org/news/2019/02/13/unified-batch- [...]
-The 1.9 release was the result of the **biggest community effort the project has experienced so far**, with the number of contributors soaring to 190 (see [The Bigger Picture](#the-bigger-picture)). For a quick overview of the upcoming work for Flink 1.10 (and beyond), have a look at the updated [roadmap](https://flink.apache.org/roadmap.html)!
+<p>The 1.9 release was the result of the <strong>biggest community effort the project has experienced so far</strong>, with the number of contributors soaring to 190 (see <a href="#the-bigger-picture">The Bigger Picture</a>). For a quick overview of the upcoming work for Flink 1.10 (and beyond), have a look at the updated <a href="https://flink.apache.org/roadmap.html">roadmap</a>!</p>
-## Integration of the Chinese-speaking community
+<h2 id="integration-of-the-chinese-speaking-community">Integration of the Chinese-speaking community</h2>
-As the number of Chinese-speaking Flink users rapidly grows, the community is working on translating resources and creating dedicated spaces for discussion to invite and include these users in the wider Flink community. Part of the ongoing work is described in [FLIP-35](https://cwiki.apache.org/confluence/display/FLINK/FLIP-35%3A+Support+Chinese+Documents+and+Website) and has resulted in:
+<p>As the number of Chinese-speaking Flink users rapidly grows, the community is working on translating resources and creating dedicated spaces for discussion to invite and include these users in the wider Flink community. Part of the ongoing work is described in <a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-35%3A+Support+Chinese+Documents+and+Website">FLIP-35</a> and has resulted in:</p>
-* A new user mailing list (user-zh@f.a.o) dedicated to Chinese-speakers.
+<ul>
+ <li>A new user mailing list (user-zh@f.a.o) dedicated to Chinese-speakers.</li>
+</ul>
<p></p>
-* A Chinese translation of the Apache Flink [website](https://flink.apache.org/zh/) and [documentation](https://ci.apache.org/projects/flink/flink-docs-master/zh/).
+<ul>
+ <li>A Chinese translation of the Apache Flink <a href="https://flink.apache.org/zh/">website</a> and <a href="https://ci.apache.org/projects/flink/flink-docs-master/zh/">documentation</a>.</li>
+</ul>
<p></p>
-* Multiple meetups organized all over China, with the biggest one reaching a whopping number of 500+ participants. Some of these meetups were also organized in collaboration with communities from other projects, like Apache Pulsar and Apache Kafka.
+<ul>
+ <li>Multiple meetups organized all over China, with the biggest one reaching a whopping number of 500+ participants. Some of these meetups were also organized in collaboration with communities from other projects, like Apache Pulsar and Apache Kafka.</li>
+</ul>
<center>
-<img src="{{ site.baseurl }}/img/blog/2019-09-05-flink-community-update/2019-09-05-flink-community-update_3.png" width="800px" alt="China Meetup"/>
+<img src="/img/blog/2019-09-05-flink-community-update/2019-09-05-flink-community-update_3.png" width="800px" alt="China Meetup" />
</center>
-In case you're interested in knowing more about this work in progress, Robert Metzger and Fabian Hueske will be diving into "Inviting Apache Flink's Chinese User Community" at the upcoming ApacheCon Europe 2019 (see [Upcoming Flink Community Events](#upcoming-flink-community-events)).
+<p>In case you’re interested in knowing more about this work in progress, Robert Metzger and Fabian Hueske will be diving into “Inviting Apache Flink’s Chinese User Community” at the upcoming ApacheCon Europe 2019 (see <a href="#upcoming-flink-community-events">Upcoming Flink Community Events</a>).</p>
-## Improving Flink's Documentation
+<h2 id="improving-flinks-documentation">Improving Flink’s Documentation</h2>
-Besides the translation effort, the community has also been working quite hard on a **Flink docs overhaul**. The main goals are to:
+<p>Besides the translation effort, the community has also been working quite hard on a <strong>Flink docs overhaul</strong>. The main goals are to:</p>
- * Organize and clean-up the structure of the docs;
- <p></p>
- * Align the content with the overall direction of the project;
- <p></p>
- * Improve the _getting-started_ material and make the content more accessible to different levels of Flink experience.
+<ul>
+ <li>Organize and clean-up the structure of the docs;</li>
+</ul>
+<p></p>
+<ul>
+ <li>Align the content with the overall direction of the project;</li>
+</ul>
+<p></p>
+<ul>
+ <li>Improve the <em>getting-started</em> material and make the content more accessible to different levels of Flink experience.</li>
+</ul>
-Given that there has been some confusion in the past regarding unclear definition of core Flink concepts, one of the first completed efforts was to introduce a [Glossary](https://ci.apache.org/projects/flink/flink-docs-release-1.9/concepts/glossary.html#glossary) in the docs. To get up to speed with the roadmap for the remainder efforts, you can refer to [FLIP-42](https://cwiki.apache.org/confluence/display/FLINK/FLIP-42%3A+Rework+Flink+Documentation) and the corresponding [umbrella Jira [...]
+<p>Given that there has been some confusion in the past regarding unclear definition of core Flink concepts, one of the first completed efforts was to introduce a <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.9/concepts/glossary.html#glossary">Glossary</a> in the docs. To get up to speed with the roadmap for the remainder efforts, you can refer to <a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-42%3A+Rework+Flink+Do [...]
-## Adjusting the Contribution Process and Experience
+<h2 id="adjusting-the-contribution-process-and-experience">Adjusting the Contribution Process and Experience</h2>
-The [guidelines](https://flink.apache.org/contributing/how-to-contribute.html) to contribute to Apache Flink have been reworked on the website, in an effort to lower the entry barrier for new contributors and reduce the overall friction in the contribution process. In addition, the Flink community discussed and adopted [bylaws](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=120731026) to help the community collaborate and coordinate more smoothly.
+<p>The <a href="https://flink.apache.org/contributing/how-to-contribute.html">guidelines</a> to contribute to Apache Flink have been reworked on the website, in an effort to lower the entry barrier for new contributors and reduce the overall friction in the contribution process. In addition, the Flink community discussed and adopted <a href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=120731026">bylaws</a> to help the [...]
-For code contributors, a [Code Style and Quality Guide](https://flink.apache.org/contributing/code-style-and-quality-preamble.html) that captures the expected standards for contributions was also added to the "Contributing" section of the Flink website.
+<p>For code contributors, a <a href="https://flink.apache.org/contributing/code-style-and-quality-preamble.html">Code Style and Quality Guide</a> that captures the expected standards for contributions was also added to the “Contributing” section of the Flink website.</p>
-It's important to stress that **contributions are not restricted to code**. Non-code contributions such as mailing list support, documentation work or organization of community events are equally as important to the development of the project and highly encouraged.
+<p>It’s important to stress that <strong>contributions are not restricted to code</strong>. Non-code contributions such as mailing list support, documentation work or organization of community events are equally as important to the development of the project and highly encouraged.</p>
-## New Committers and PMC Members
+<h2 id="new-committers-and-pmc-members">New Committers and PMC Members</h2>
-The Apache Flink community has welcomed **5 new Committers** and **4 PMC (Project Management Committee) Members** in 2019, so far:
+<p>The Apache Flink community has welcomed <strong>5 new Committers</strong> and <strong>4 PMC (Project Management Committee) Members</strong> in 2019, so far:</p>
-### New PMC Members
- Jincheng Sun, Kete (Kurt) Young, Kostas Kloudas, Thomas Weise
+<h3 id="new-pmc-members">New PMC Members</h3>
+<div class="highlight"><pre><code>Jincheng Sun, Kete (Kurt) Young, Kostas Kloudas, Thomas Weise
+</code></pre></div>
-### New Committers
- Andrey Zagrebin, Hequn, Jiangjie (Becket) Qin, Rong Rong, Zhijiang Wang
+<h3 id="new-committers">New Committers</h3>
+<div class="highlight"><pre><code>Andrey Zagrebin, Hequn, Jiangjie (Becket) Qin, Rong Rong, Zhijiang Wang
+</code></pre></div>
-Congratulations and thank you for your hardworking commitment to Flink!
+<p>Congratulations and thank you for your hardworking commitment to Flink!</p>
-# The Bigger Picture
+<h1 id="the-bigger-picture">The Bigger Picture</h1>
-Flink continues to push the boundaries of (stream) data processing, and the community is proud to see an ever-increasingly diverse set of contributors, users and technologies join the ecosystem.
+<p>Flink continues to push the boundaries of (stream) data processing, and the community is proud to see an ever-increasingly diverse set of contributors, users and technologies join the ecosystem.</p>
-In the timeframe of three releases, the project jumped from **112 to 190 contributors**, also doubling down on the number of requested changes and improvements. To top it off, the Flink GitHub repository recently reached the milestone of **10k stars**, all the way up from the incubation days in 2014.
+<p>In the timeframe of three releases, the project jumped from <strong>112 to 190 contributors</strong>, also doubling down on the number of requested changes and improvements. To top it off, the Flink GitHub repository recently reached the milestone of <strong>10k stars</strong>, all the way up from the incubation days in 2014.</p>
<center>
-<img src="{{ site.baseurl }}/img/blog/2019-09-05-flink-community-update/2019-09-05-flink-community-update_1.png" width="1000px" alt="GitHub"/>
+<img src="/img/blog/2019-09-05-flink-community-update/2019-09-05-flink-community-update_1.png" width="1000px" alt="GitHub" />
</center>
-The activity across the user@ and dev@<sup>1</sup> mailing lists shows a healthy heartbeat, and the gradual ramp up of user-zh@ suggests that this was a well-received community effort. Looking at the numbers for the same period in 2018, the dev@ mailing list has seen the biggest surge in activity, with an average growth of **2.5x in the number of messages and distinct users** — a great reflection of the hyperactive pace of development of the Flink codebase.
+<p>The activity across the user@ and dev@<sup>1</sup> mailing lists shows a healthy heartbeat, and the gradual ramp up of user-zh@ suggests that this was a well-received community effort. Looking at the numbers for the same period in 2018, the dev@ mailing list has seen the biggest surge in activity, with an average growth of <strong>2.5x in the number of messages and distinct users</strong> — a great reflection of the hyperactive pace of development of the [...]
-<img style="float: right;" src="{{ site.baseurl }}/img/blog/2019-09-05-flink-community-update/2019-09-05-flink-community-update_2.png" width="420px" alt="Mailing Lists"/>
+<p><img style="float: right;" src="/img/blog/2019-09-05-flink-community-update/2019-09-05-flink-community-update_2.png" width="420px" alt="Mailing Lists" /></p>
-In support of these observations, the report for the financial year of 2019 from the Apache Software Foundation (ASF) features Flink as one of the most thriving open source projects, with mentions for:
+<p>In support of these observations, the report for the financial year of 2019 from the Apache Software Foundation (ASF) features Flink as one of the most thriving open source projects, with mentions for:</p>
-* Most Active Visits and Downloads
+<ul>
+ <li>Most Active Visits and Downloads</li>
+</ul>
<p></p>
-* Most Active Sources: Visits
+<ul>
+ <li>Most Active Sources: Visits</li>
+</ul>
<p></p>
-* Most Active Sources: Clones
+<ul>
+ <li>Most Active Sources: Clones</li>
+</ul>
<p></p>
-* Top Repositories by Number of Commits
+<ul>
+ <li>Top Repositories by Number of Commits</li>
+</ul>
<p></p>
-* Top Most Active Apache Mailing Lists (user@ and dev@)
+<ul>
+ <li>Top Most Active Apache Mailing Lists (user@ and dev@)</li>
+</ul>
-Hats off to our fellows at Apache Beam for an astounding year, too! For more detailed insights, check the [full report](https://s3.amazonaws.com/files-dist/AnnualReports/FY2018%20Annual%20Report.pdf).
+<p>Hats off to our fellows at Apache Beam for an astounding year, too! For more detailed insights, check the <a href="https://s3.amazonaws.com/files-dist/AnnualReports/FY2018%20Annual%20Report.pdf">full report</a>.</p>
<p></p>
-<sup>1. Excluding messages from "jira@apache.org".</sup>
+<p><sup>1. Excluding messages from “jira@apache.org”.</sup></p>
-# Upcoming Events
+<h1 id="upcoming-events">Upcoming Events</h1>
-As the conference and meetup season ramps up again, here are some events to keep an eye out for talks about Flink and opportunities to mingle with the wider stream processing community.
+<p>As the conference and meetup season ramps up again, here are some events to keep an eye out for talks about Flink and opportunities to mingle with the wider stream processing community.</p>
-### North America
+<h3 id="north-america">North America</h3>
-* [Conference] **[Strata Data Conference 2019](https://conferences.oreilly.com/strata/strata-ny)**, September 23-26, New York, USA
- <p></p>
-* [Meetup] **[Apache Flink Bay Area Meetup](https://www.meetup.com/Bay-Area-Apache-Flink-Meetup/events/262680261/)**, September 24, San Francisco, USA
- <p></p>
-* [Conference] **[Scale By The Bay 2019](https://www.meetup.com/Bay-Area-Apache-Flink-Meetup/events/262680261/)**, November 13-15, San Francisco, USA
-
-### Europe
+<ul>
+ <li>[Conference] <strong><a href="https://conferences.oreilly.com/strata/strata-ny">Strata Data Conference 2019</a></strong>, September 23-26, New York, USA
+ <p></p>
+ </li>
+ <li>[Meetup] <strong><a href="https://www.meetup.com/Bay-Area-Apache-Flink-Meetup/events/262680261/">Apache Flink Bay Area Meetup</a></strong>, September 24, San Francisco, USA
+ <p></p>
+ </li>
+ <li>[Conference] <strong><a href="https://www.meetup.com/Bay-Area-Apache-Flink-Meetup/events/262680261/">Scale By The Bay 2019</a></strong>, November 13-15, San Francisco, USA</li>
+</ul>
-* [Meetup] **[Apache Flink London Meetup](https://www.meetup.com/Apache-Flink-London-Meetup/events/264123672)**, September 23, London, UK
- <p></p>
-* [Conference] **[Flink Forward Europe 2019](https://europe-2019.flink-forward.org)**, October 7-9, Berlin, Germany
- <p></p>
- * The next edition of Flink Forward Europe is around the corner and the [program](https://europe-2019.flink-forward.org/conference-program) has been announced, featuring 70+ talks as well as panel discussions and interactive "Ask Me Anything" sessions with core Flink committers. If you're looking to learn more about Flink and share your experience with other community members, there really is [no better place]((https://vimeo.com/296403091)) than Flink Forward!
+<h3 id="europe">Europe</h3>
- * **Note:** if you are a **committer for any Apache project**, you can **get a free ticket** by registering with your Apache email address and using the discount code: *FFEU19-ApacheCommitter*.
+<ul>
+ <li>[Meetup] <strong><a href="https://www.meetup.com/Apache-Flink-London-Meetup/events/264123672">Apache Flink London Meetup</a></strong>, September 23, London, UK
+ <p></p>
+ </li>
+ <li>[Conference] <strong><a href="https://europe-2019.flink-forward.org">Flink Forward Europe 2019</a></strong>, October 7-9, Berlin, Germany
+ <p></p>
+ <ul>
+ <li>
+ <p>The next edition of Flink Forward Europe is around the corner and the <a href="https://europe-2019.flink-forward.org/conference-program">program</a> has been announced, featuring 70+ talks as well as panel discussions and interactive “Ask Me Anything” sessions with core Flink committers. If you’re looking to learn more about Flink and share your experience with other community members, there really is <a href="(https://vimeo.com/296403091)&q [...]
+ </li>
+ <li>
+ <p><strong>Note:</strong> if you are a <strong>committer for any Apache project</strong>, you can <strong>get a free ticket</strong> by registering with your Apache email address and using the discount code: <em>FFEU19-ApacheCommitter</em>.</p>
+ </li>
+ </ul>
+ </li>
+</ul>
<p></p>
-* [Conference] **[ApacheCon Berlin 2019](https://aceu19.apachecon.com/)**, October 22-24, Berlin, Germany
+<ul>
+ <li>[Conference] <strong><a href="https://aceu19.apachecon.com/">ApacheCon Berlin 2019</a></strong>, October 22-24, Berlin, Germany</li>
+</ul>
<p></p>
-* [Conference] **[Data2Day 2019](https://www.data2day.de/)**, October 22-24, Ludwigshafen, Germany
+<ul>
+ <li>[Conference] <strong><a href="https://www.data2day.de/">Data2Day 2019</a></strong>, October 22-24, Ludwigshafen, Germany</li>
+</ul>
<p></p>
-* [Conference] **[Big Data Tech Warsaw 2020](https://bigdatatechwarsaw.eu)**, February 7, Warsaw, Poland
- <p></p>
- * The Call For Presentations (CFP) is now [open](https://bigdatatechwarsaw.eu/cfp/).
+<ul>
+ <li>[Conference] <strong><a href="https://bigdatatechwarsaw.eu">Big Data Tech Warsaw 2020</a></strong>, February 7, Warsaw, Poland
+ <p></p>
+ <ul>
+ <li>The Call For Presentations (CFP) is now <a href="https://bigdatatechwarsaw.eu/cfp/">open</a>.</li>
+ </ul>
+ </li>
+</ul>
-### Asia
+<h3 id="asia">Asia</h3>
-* [Conference] **[Flink Forward Asia 2019](https://m.aliyun.com/markets/aliyun/developer/ffa2019)**, November 28-30, Beijing, China
- <p></p>
- * The second edition of Flink Forward Asia is also happening later this year, in Beijing, and the CFP is [open](https://developer.aliyun.com/special/ffa2019) until September 20.
+<ul>
+ <li>[Conference] <strong><a href="https://m.aliyun.com/markets/aliyun/developer/ffa2019">Flink Forward Asia 2019</a></strong>, November 28-30, Beijing, China
+ <p></p>
+ <ul>
+ <li>The second edition of Flink Forward Asia is also happening later this year, in Beijing, and the CFP is <a href="https://developer.aliyun.com/special/ffa2019">open</a> until September 20.</li>
+ </ul>
+ </li>
+</ul>
-If you'd like to keep a closer eye on what’s happening in the community, subscribe to the [community mailing list](https://flink.apache.org/community.html#mailing-lists) to get fine-grained weekly updates, upcoming event announcements and more. Also, please reach out if you're interested in organizing or being part of Flink events in your area!</description>
-<pubDate>Tue, 10 Sep 2019 12:00:00 +0000</pubDate>
+<p>If you’d like to keep a closer eye on what’s happening in the community, subscribe to the <a href="https://flink.apache.org/community.html#mailing-lists">community mailing list</a> to get fine-grained weekly updates, upcoming event announcements and more. Also, please reach out if you’re interested in organizing or being part of Flink events in your area!</p>
+</description>
+<pubDate>Tue, 10 Sep 2019 14:00:00 +0200</pubDate>
<link>https://flink.apache.org/news/2019/09/10/community-update.html</link>
<guid isPermaLink="true">/news/2019/09/10/community-update.html</guid>
</item>
<item>
<title>Apache Flink 1.9.0 Release Announcement</title>
-<description>The Apache Flink community is proud to announce the release of Apache Flink
-1.9.0.
+<description><p>The Apache Flink community is proud to announce the release of Apache Flink
+1.9.0.</p>
-The Apache Flink project's goal is to develop a stream processing system to
+<p>The Apache Flink project’s goal is to develop a stream processing system to
unify and power many forms of real-time and offline data processing
applications as well as event-driven applications. In this release, we have
made a huge step forward in that effort, by integrating Flink’s stream and
-batch processing capabilities under a single, unified runtime.
+batch processing capabilities under a single, unified runtime.</p>
-Significant features on this path are batch-style recovery for batch jobs and
+<p>Significant features on this path are batch-style recovery for batch jobs and
a preview of the new Blink-based query engine for Table API and SQL queries.
We are also excited to announce the availability of the State Processor API,
which is one of the most frequently requested features and enables users to
read and write savepoints with Flink DataSet jobs. Finally, Flink 1.9 includes
a reworked WebUI and previews of Flink’s new Python Table API and its
-integration with the Apache Hive ecosystem.
+integration with the Apache Hive ecosystem.</p>
-This blog post describes all major new features and improvements, important
+<p>This blog post describes all major new features and improvements, important
changes to be aware of and what to expect moving forward. For more details,
-check the [complete release
-changelog](https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12344601).
+check the <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&amp;version=12344601">complete release
+changelog</a>.</p>
-The binary distribution and source artifacts for this release are now
-available via the [Downloads](https://flink.apache.org/downloads.html) page of
+<p>The binary distribution and source artifacts for this release are now
+available via the <a href="https://flink.apache.org/downloads.html">Downloads</a> page of
the Flink project, along with the updated
-[documentation](https://ci.apache.org/projects/flink/flink-docs-release-1.9/).
+<a href="https://ci.apache.org/projects/flink/flink-docs-release-1.9/">documentation</a>.
Flink 1.9 is API-compatible with previous 1.x releases for APIs annotated with
-the `@Public` annotation.
-
-Please feel encouraged to download the release and share your thoughts with
-the community through the Flink [mailing
-lists](https://flink.apache.org/community.html#mailing-lists) or
-[JIRA](https://issues.apache.org/jira/projects/FLINK/summary). As always,
-feedback is very much appreciated!
-
-
-{% toc %}
-
+the <code>@Public</code> annotation.</p>
+
+<p>Please feel encouraged to download the release and share your thoughts with
+the community through the Flink <a href="https://flink.apache.org/community.html#mailing-lists">mailing
+lists</a> or
+<a href="https://issues.apache.org/jira/projects/FLINK/summary">JIRA</a>. As always,
+feedback is very much appreciated!</p>
+
+<div class="page-toc">
+<ul id="markdown-toc">
+ <li><a href="#new-features-and-improvements" id="markdown-toc-new-features-and-improvements">New Features and Improvements</a> <ul>
+ <li><a href="#fine-grained-batch-recovery-flip-1" id="markdown-toc-fine-grained-batch-recovery-flip-1">Fine-grained Batch Recovery (FLIP-1)</a></li>
+ <li><a href="#state-processor-api-flip-43" id="markdown-toc-state-processor-api-flip-43">State Processor API (FLIP-43)</a></li>
+ <li><a href="#stop-with-savepoint-flip-34" id="markdown-toc-stop-with-savepoint-flip-34">Stop-with-Savepoint (FLIP-34)</a></li>
+ <li><a href="#flink-webui-rework" id="markdown-toc-flink-webui-rework">Flink WebUI Rework</a></li>
+ <li><a href="#preview-of-the-new-blink-sql-query-processor" id="markdown-toc-preview-of-the-new-blink-sql-query-processor">Preview of the new Blink SQL Query Processor</a></li>
+ <li><a href="#preview-of-full-hive-integration-flink-10556" id="markdown-toc-preview-of-full-hive-integration-flink-10556">Preview of Full Hive Integration (FLINK-10556)</a></li>
+ <li><a href="#preview-of-the-new-python-table-api-flip-38" id="markdown-toc-preview-of-the-new-python-table-api-flip-38">Preview of the new Python Table API (FLIP-38)</a></li>
+ </ul>
+ </li>
+ <li><a href="#important-changes" id="markdown-toc-important-changes">Important Changes</a></li>
+ <li><a href="#release-notes" id="markdown-toc-release-notes">Release Notes</a></li>
+ <li><a href="#list-of-contributors" id="markdown-toc-list-of-contributors">List of Contributors</a></li>
+</ul>
-## New Features and Improvements
+</div>
+<h2 id="new-features-and-improvements">New Features and Improvements</h2>
-### Fine-grained Batch Recovery (FLIP-1)
+<h3 id="fine-grained-batch-recovery-flip-1">Fine-grained Batch Recovery (FLIP-1)</h3>
-The time to recover a batch (DataSet, Table API and SQL) job from a task
+<p>The time to recover a batch (DataSet, Table API and SQL) job from a task
failure was significantly reduced. Until Flink 1.9, task failures in batch
jobs were recovered by canceling all tasks and restarting the whole job, i.e,
the job was started from scratch and all progress was voided. With this
release, Flink can be configured to limit the recovery to only those tasks
-that are in the same **failover region**. A failover region is the set of
+that are in the same <strong>failover region</strong>. A failover region is the set of
tasks that are connected via pipelined data exchanges. Hence, the
batch-shuffle connections of a job define the boundaries of its failover
regions. More details are available in
-[FLIP-1](https://cwiki.apache.org/confluence/display/FLINK/FLIP-1+%3A+Fine+Grained+Recovery+from+Task+Failures).
-
+<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-1+%3A+Fine+Grained+Recovery+from+Task+Failures">FLIP-1</a>.
+<img src="/img/blog/release-19-flip1.png" alt="alt_text" title="Fine-grained Batch
+Recovery" /></p>
-To use this new failover strategy, you need to do the following
-settings:
+<p>To use this new failover strategy, you need to do the following
+settings:</p>
- * Make sure you have the entry `jobmanager.execution.failover-strategy:
- region` in your `flink-conf.yaml`.
+<ul>
+ <li>Make sure you have the entry <code>jobmanager.execution.failover-strategy:
+region</code> in your <code>flink-conf.yaml</code>.</li>
+</ul>
-**Note:** The configuration of the 1.9 distribution has that entry by default,
+<p><strong>Note:</strong> The configuration of the 1.9 distribution has that entry by default,
but when reusing a configuration file from previous setups, you have to add
- it manually.
+ it manually.</p>
-Moreover, you need to set the `ExecutionMode` of batch jobs in the
-`ExecutionConfig` to `BATCH` to configure that data shuffles are not pipelined
-and jobs have more than one failover region.
+<p>Moreover, you need to set the <code>ExecutionMode</code> of batch jobs in the
+<code>ExecutionConfig</code> to <code>BATCH</code> to configure that data shuffles are not pipelined
+and jobs have more than one failover region.</p>
-The "Region" failover strategy also improves the recovery of “embarrassingly
+<p>The “Region” failover strategy also improves the recovery of “embarrassingly
parallel” streaming jobs, i.e., jobs without any shuffle like keyBy() or
rebalance. When such a job is recovered, only the tasks of the affected
pipeline (failover region) are restarted. For all other streaming jobs, the
-recovery behavior is the same as in prior Flink versions.
+recovery behavior is the same as in prior Flink versions.</p>
+<h3 id="state-processor-api-flip-43">State Processor API (FLIP-43)</h3>
-### State Processor API (FLIP-43)
-
-Up to Flink 1.9, accessing the state of a job from the outside was limited to
-the (still) experimental [Queryable
-State](https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/queryable_state.html).
+<p>Up to Flink 1.9, accessing the state of a job from the outside was limited to
+the (still) experimental <a href="https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/queryable_state.html">Queryable
+State</a>.
This release introduces a new, powerful library to read, write and modify
-state snapshots using the batch DataSet API. In practice, this means:
-
- * Flink job state can be bootstrapped by reading data from external systems,
- such as external databases, and converting it into a savepoint.
- * State in savepoints can be queried using any of Flink’s batch APIs
- (DataSet, Table, SQL), for example to analyze relevant state patterns or
- check for discrepancies in state that can support application auditing or
- troubleshooting.
- * The schema of state in savepoints can be migrated offline, compared to the
- previous approach requiring online migration on schema access.
- * Invalid data in savepoints can be identified and corrected.
-
-The new State Processor API covers all variations of snapshots: savepoints,
-full checkpoints and incremental checkpoints. More details are available in
-[FLIP-43](https://cwiki.apache.org/confluence/display/FLINK/FLIP-43%3A+State+Processor+API)
+state snapshots using the batch DataSet API. In practice, this means:</p>
+
+<ul>
+ <li>Flink job state can be bootstrapped by reading data from external systems,
+such as external databases, and converting it into a savepoint.</li>
+ <li>State in savepoints can be queried using any of Flink’s batch APIs
+(DataSet, Table, SQL), for example to analyze relevant state patterns or
+check for discrepancies in state that can support application auditing or
+troubleshooting.</li>
+ <li>The schema of state in savepoints can be migrated offline, compared to the
+previous approach requiring online migration on schema access.</li>
+ <li>Invalid data in savepoints can be identified and corrected.</li>
+</ul>
+<p>The new State Processor API covers all variations of snapshots: savepoints,
+full checkpoints and incremental checkpoints. More details are available in
+<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-43%3A+State+Processor+API">FLIP-43</a></p>
-### Stop-with-Savepoint (FLIP-34)
+<h3 id="stop-with-savepoint-flip-34">Stop-with-Savepoint (FLIP-34)</h3>
-[Cancelling with a
-savepoint](https://ci.apache.org/projects/flink/flink-docs-stable/ops/state/savepoints.html#operations)
+<p><a href="https://ci.apache.org/projects/flink/flink-docs-stable/ops/state/savepoints.html#operations">Cancelling with a
+savepoint</a>
is a common operation for stopping/restarting, forking or updating Flink jobs.
However, the existing implementation did not guarantee output persistence to
external storage systems for exactly-once sinks. To improve the end-to-end
-semantics when stopping a job, Flink 1.9 introduces a new `SUSPEND` mode to
+semantics when stopping a job, Flink 1.9 introduces a new <code>SUSPEND</code> mode to
stop a job with a savepoint that is consistent with the emitted data.
-You can suspend a job with Flink’s CLI client as follows:
-
-```
-bin/flink stop -p [:targetDirectory] :jobId
-```
-
-The final job state is set to `FINISHED` on success, allowing
-users to detect failures of the requested operation.
+You can suspend a job with Flink’s CLI client as follows:</p>
-More details are available in
-[FLIP-34](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=103090212)
+<div class="highlight"><pre><code>bin/flink stop -p [:targetDirectory] :jobId
+</code></pre></div>
+<p>The final job state is set to <code>FINISHED</code> on success, allowing
+users to detect failures of the requested operation.</p>
+<p>More details are available in
+<a href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=103090212">FLIP-34</a></p>
-### Flink WebUI Rework
+<h3 id="flink-webui-rework">Flink WebUI Rework</h3>
-After a
-[discussion](http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Change-underlying-Frontend-Architecture-for-Flink-Web-Dashboard-td24902.html)
+<p>After a
+<a href="http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Change-underlying-Frontend-Architecture-for-Flink-Web-Dashboard-td24902.html">discussion</a>
about modernizing the internals of Flink’s WebUI, this component was
reconstructed using the latest stable version of Angular — basically, a bump
from Angular 1.x to 7.x. The redesigned version is the default in 1.9.0,
-however there is a link to switch to the old WebUI.
+however there is a link to switch to the old WebUI.</p>
-<div class="row"> <div class="col-sm-6"> <span><img class="thumbnail"
- src="{{site.baseurl}}/img/blog/release-19-web1.png" /></span> </div> <div
- class="col-sm-6"> <span><img class="thumbnail"
- src="{{site.baseurl}}/img/blog/release-19-web2.png" /></span> </div>
+<div class="row"> <div class="col-sm-6"> <span><img class="thumbnail" src="/img/blog/release-19-web1.png" /></span> </div> <div class="col-sm-6"> <span><img class="thumbnail" src="/img/blog/release-19-web2.png" /></span> </div>
</div>
-**Note:** Moving forward, feature parity for the old version of the WebUI
-will not be guaranteed.
+<p><strong>Note:</strong> Moving forward, feature parity for the old version of the WebUI
+will not be guaranteed.</p>
+<h3 id="preview-of-the-new-blink-sql-query-processor">Preview of the new Blink SQL Query Processor</h3>
-### Preview of the new Blink SQL Query Processor
-
-Following the [donation of
-Blink]({{site.baseurl}}/news/2019/02/13/unified-batch-streaming-blink.html) to
+<p>Following the <a href="/news/2019/02/13/unified-batch-streaming-blink.html">donation of
+Blink</a> to
Apache Flink, the community worked on integrating Blink’s query optimizer and
runtime for the Table API and SQL. As a first step, we refactored the
-monolithic `flink-table` module into smaller modules
-([FLIP-32](https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions)).
+monolithic <code>flink-table</code> module into smaller modules
+(<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions">FLIP-32</a>).
This resulted in a clear separation of and well-defined interfaces between the
-Java and Scala API modules and the optimizer and runtime modules.
+Java and Scala API modules and the optimizer and runtime modules.</p>
-<span><img style="width:50%"
-src="{{site.baseurl}}/img/blog/release-19-stack.png" /></span>
+<p><span><img style="width:50%" src="/img/blog/release-19-stack.png" /></span></p>
-Next, we extended Blink’s planner to implement the new optimizer interface
+<p>Next, we extended Blink’s planner to implement the new optimizer interface
such that there are now two pluggable query processors to execute Table API
and SQL statements: the pre-1.9 Flink processor and the new Blink-based query
processor. The Blink-based query processor offers better SQL coverage (full TPC-H
@@ -2975,161 +3030,165 @@ code-generation, and tuned operator implementations.
The Blink-based query processor also provides a more powerful streaming runner,
with some new features (e.g. dimension table join, TopN, deduplication) and
optimizations to solve data-skew in aggregation and more useful built-in
-functions.
+functions.</p>
-**Note:** The semantics and set of supported operations of the query
-processors are mostly, but not fully aligned.
+<p><strong>Note:</strong> The semantics and set of supported operations of the query
+processors are mostly, but not fully aligned.</p>
-However, the integration of Blink’s query processor is not fully completed
+<p>However, the integration of Blink’s query processor is not fully completed
yet. Therefore, the pre-1.9 Flink processor is still the default processor in
Flink 1.9 and recommended for production settings. You can enable the Blink
-processor by configuring it via the `EnvironmentSettings` when creating a
-`TableEnvironment`. The selected processor must be on the classpath of the
+processor by configuring it via the <code>EnvironmentSettings</code> when creating a
+<code>TableEnvironment</code>. The selected processor must be on the classpath of the
executing Java process. For cluster setups, both query processors are
automatically loaded with the default configuration. When running a query from
-your IDE you need to explicitly [add a planner
-dependency](https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/table/#table-program-dependencies)
-to your project.
-
-
-#### **Other Improvements to the Table API and SQL**
-
-Besides the exciting progress around the Blink planner, the community worked
-on a whole set of other improvements to these interfaces, including:
-
- * **Scala-free Table API and SQL for Java users
- ([FLIP-32](https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions))**
-
- As part of the refactoring and splitting of the flink-table module, two
- separate API modules for Java and Scala were created. For Scala users,
- nothing really changes, but Java users can use the Table API and/or SQL now
- without pulling in a Scala dependency.
-
- * **Rework of the Table API Type System**
- **([FLIP-37](https://cwiki.apache.org/confluence/display/FLINK/FLIP-37%3A+Rework+of+the+Table+API+Type+System))**
-
- The community implemented a [new data type
- system](https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/table/types.html#data-types)
- to detach the Table API from Flink’s
- [TypeInformation](https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/types_serialization.html#flinks-typeinformation-class)
- class and improve its compliance with the SQL standard. This is still a
- work in progress and expected to be completed in the next release. In
- Flink 1.9, UDFs are―among other things―not ported to the new type system
- yet.
-
- * **Multi-column and Multi-row Transformations for Table API**
- **([FLIP-29](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=97552739))**
-
- The functionality of the Table API was extended with a set of
- transformations that support multi-row and/or multi-column inputs and
- outputs. These transformations significantly ease the implementation of
- processing logic that would be cumbersome to implement with relational
- operators.
-
- * **New, Unified Catalog APIs**
- **([FLIP-30](https://cwiki.apache.org/confluence/display/FLINK/FLIP-30%3A+Unified+Catalog+APIs))**
-
- We reworked the catalog APIs to store metadata and unified the handling of
- internal and external catalogs. This effort was mainly initiated as a
- prerequisite for the Hive integration (see below), but improves the overall
- convenience of managing catalog metadata in Flink. Besides improving the
- catalog interfaces, we also extended their functionality. Previously table
- definitions for Table API or SQL queries were volatile. With Flink 1.9, the
- metadata of tables which are registered with a SQL DDL statement can be
- persisted in a catalog. This means you can add a table that is backed by a
- Kafka topic to a Metastore catalog and from then on query this table
- whenever your catalog is connected to Metastore.
-
- * **DDL Support in the SQL API
- ([FLINK-10232](https://issues.apache.org/jira/browse/FLINK-10232))**
-
- Up to this point, Flink SQL only supported DML statements (e.g. `SELECT`,
- `INSERT`). External tables (table sources and sinks) had to be registered
- via Java/Scala code or configuration files. For 1.9, we added support for
- SQL DDL statements to register and remove tables and views (`CREATE TABLE,
- DROP TABLE)`. However, we did not add
- stream-specific syntax extensions to define timestamp extraction and
- watermark generation, yet. Full support for streaming use cases is planned
- for the next release.
-
-
-### Preview of Full Hive Integration (FLINK-10556)
-
-Apache Hive is widely used in Hadoop’s ecosystem to store and query large
+your IDE you need to explicitly <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/table/#table-program-dependencies">add a planner
+dependency</a>
+to your project.</p>
+
+<h4 id="other-improvements-to-the-table-api-and-sql"><strong>Other Improvements to the Table API and SQL</strong></h4>
+
+<p>Besides the exciting progress around the Blink planner, the community worked
+on a whole set of other improvements to these interfaces, including:</p>
+
+<ul>
+ <li>
+ <p><strong>Scala-free Table API and SQL for Java users
+(<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions">FLIP-32</a>)</strong></p>
+
+ <p>As part of the refactoring and splitting of the flink-table module, two
+separate API modules for Java and Scala were created. For Scala users,
+nothing really changes, but Java users can use the Table API and/or SQL now
+without pulling in a Scala dependency.</p>
+ </li>
+ <li>
+ <p><strong>Rework of the Table API Type System</strong>
+<strong>(<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-37%3A+Rework+of+the+Table+API+Type+System">FLIP-37</a>)</strong></p>
+
+ <p>The community implemented a <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/table/types.html#data-types">new data type
+system</a>
+to detach the Table API from Flink’s
+<a href="https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/types_serialization.html#flinks-typeinformation-class">TypeInformation</a>
+class and improve its compliance with the SQL standard. This is still a
+work in progress and expected to be completed in the next release. In
+Flink 1.9, UDFs are―among other things―not ported to the new type system
+yet.</p>
+ </li>
+ <li>
+ <p><strong>Multi-column and Multi-row Transformations for Table API</strong>
+<strong>(<a href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=97552739">FLIP-29</a>)</strong></p>
+
+ <p>The functionality of the Table API was extended with a set of
+transformations that support multi-row and/or multi-column inputs and
+outputs. These transformations significantly ease the implementation of
+processing logic that would be cumbersome to implement with relational
+operators.</p>
+ </li>
+ <li>
+ <p><strong>New, Unified Catalog APIs</strong>
+<strong>(<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-30%3A+Unified+Catalog+APIs">FLIP-30</a>)</strong></p>
+
+ <p>We reworked the catalog APIs to store metadata and unified the handling of
+internal and external catalogs. This effort was mainly initiated as a
+prerequisite for the Hive integration (see below), but improves the overall
+convenience of managing catalog metadata in Flink. Besides improving the
+catalog interfaces, we also extended their functionality. Previously table
+definitions for Table API or SQL queries were volatile. With Flink 1.9, the
+metadata of tables which are registered with a SQL DDL statement can be
+persisted in a catalog. This means you can add a table that is backed by a
+Kafka topic to a Metastore catalog and from then on query this table
+whenever your catalog is connected to Metastore.</p>
+ </li>
+ <li>
+ <p><strong>DDL Support in the SQL API
+(<a href="https://issues.apache.org/jira/browse/FLINK-10232">FLINK-10232</a>)</strong></p>
+
+ <p>Up to this point, Flink SQL only supported DML statements (e.g. <code>SELECT</code>,
+<code>INSERT</code>). External tables (table sources and sinks) had to be registered
+via Java/Scala code or configuration files. For 1.9, we added support for
+SQL DDL statements to register and remove tables and views (<code>CREATE TABLE,
+DROP TABLE)</code>. However, we did not add
+stream-specific syntax extensions to define timestamp extraction and
+watermark generation, yet. Full support for streaming use cases is planned
+for the next release.</p>
+ </li>
+</ul>
+
+<h3 id="preview-of-full-hive-integration-flink-10556">Preview of Full Hive Integration (FLINK-10556)</h3>
+
+<p>Apache Hive is widely used in Hadoop’s ecosystem to store and query large
amounts of structured data. Besides being a query processor, Hive features a
catalog called Metastore to manage and organize large datasets. A common
integration point for query processors is to integrate with Hive’s Metastore
-in order to be able to tap into the data managed by Hive.
+in order to be able to tap into the data managed by Hive.</p>
-Recently, the community started implementing an external catalog for Flink’s
+<p>Recently, the community started implementing an external catalog for Flink’s
Table API and SQL that connects to Hive’s Metastore. In Flink 1.9, users will
be able to query and process all data that is stored in Hive. As described
earlier, you will also be able to persist metadata of Flink tables in Metastore.
Moreover, the Hive integration includes support to use Hive’s UDFs in Flink
Table API or SQL queries. More details are available in
-[FLINK-10556](https://issues.apache.org/jira/browse/FLINK-10556).
+<a href="https://issues.apache.org/jira/browse/FLINK-10556">FLINK-10556</a>.</p>
-While, previously, table definitions for Table API or SQL queries were always
+<p>While, previously, table definitions for Table API or SQL queries were always
volatile, the new catalog connector additionally allows persisting a table in
Metastore that is created with a SQL DDL statement (see above). This means
that you connect to Metastore and register a table that is, for example,
backed by a Kafka topic. From now on, you can query that table whenever your
-catalog is connected to Metastore.
+catalog is connected to Metastore.</p>
-Please note that the Hive support in Flink 1.9 is experimental. We are
+<p>Please note that the Hive support in Flink 1.9 is experimental. We are
planning to stabilize these features for the next release and are looking
-forward to your feedback.
+forward to your feedback.</p>
+<h3 id="preview-of-the-new-python-table-api-flip-38">Preview of the new Python Table API (FLIP-38)</h3>
-### Preview of the new Python Table API (FLIP-38)
-
-This release also introduces a first version of a Python Table API
-([FLIP-38](https://cwiki.apache.org/confluence/display/FLINK/FLIP-38%3A+Python+Table+API)).
+<p>This release also introduces a first version of a Python Table API
+(<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-38%3A+Python+Table+API">FLIP-38</a>).
This marks the start towards our goal of bringing
full-fledged Python support to Flink. The feature was designed as a slim
Python API wrapper around the Table API, basically translating Python Table
API method calls into Java Table API calls. In the initial version that ships
with Flink 1.9, the Python Table API does not support UDFs yet, but just
standard relational operations. Support for UDFs implemented in Python is on
-the roadmap for future releases.
-
-If you’d like to try the new Python API, you have to manually [install
-PyFlink](https://ci.apache.org/projects/flink/flink-docs-release-1.9/flinkDev/building.html#build-pyflink).
-From there, you can have a look at [this
-walkthrough](https://ci.apache.org/projects/flink/flink-docs-release-1.9/tutorials/python_table_api.html)
-or explore it on your own. The [community is currently
-working](http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-Publish-the-PyFlink-into-PyPI-td31201.html)
-on preparing a `pyflink` Python package that will be made available for
-installation via `pip`.
+the roadmap for future releases.</p>
+<p>If you’d like to try the new Python API, you have to manually <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.9/flinkDev/building.html#build-pyflink">install
+PyFlink</a>.
+From there, you can have a look at <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.9/tutorials/python_table_api.html">this
+walkthrough</a>
+or explore it on your own. The <a href="http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-Publish-the-PyFlink-into-PyPI-td31201.html">community is currently
+working</a>
+on preparing a <code>pyflink</code> Python package that will be made available for
+installation via <code>pip</code>.</p>
-## Important Changes
-
- * The Table API and SQL are now part of the default configuration of the
- Flink distribution. Before, the Table API and SQL had to be enabled by
- moving the corresponding JAR file from ./opt to ./lib.
- * The machine learning library (flink-ml) has been removed in preparation for
- [FLIP-39](https://docs.google.com/document/d/1StObo1DLp8iiy0rbukx8kwAJb0BwDZrQrMWub3DzsEo/edit).
- * The old DataSet and DataStream Python APIs have been removed in favor of
- [FLIP-38](https://cwiki.apache.org/confluence/display/FLINK/FLIP-38%3A+Python+Table+API).
- * Flink can be compiled and run on Java 9. Note that certain components
- interacting with external systems (connectors, filesystems, reporters) may
- not work since the respective projects may have skipped Java 9 support.
+<h2 id="important-changes">Important Changes</h2>
+<ul>
+ <li>The Table API and SQL are now part of the default configuration of the
+Flink distribution. Before, the Table API and SQL had to be enabled by
+moving the corresponding JAR file from ./opt to ./lib.</li>
+ <li>The machine learning library (flink-ml) has been removed in preparation for
+<a href="https://docs.google.com/document/d/1StObo1DLp8iiy0rbukx8kwAJb0BwDZrQrMWub3DzsEo/edit">FLIP-39</a>.</li>
+ <li>The old DataSet and DataStream Python APIs have been removed in favor of
+<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-38%3A+Python+Table+API">FLIP-38</a>.</li>
+ <li>Flink can be compiled and run on Java 9. Note that certain components
+interacting with external systems (connectors, filesystems, reporters) may
+not work since the respective projects may have skipped Java 9 support.</li>
+</ul>
-## Release Notes
+<h2 id="release-notes">Release Notes</h2>
-Please review the [release
-notes](https://ci.apache.org/projects/flink/flink-docs-release-1.9/release-notes/flink-1.9.html)
+<p>Please review the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.9/release-notes/flink-1.9.html">release
+notes</a>
for a more detailed list of changes and new features if you plan to upgrade
-your Flink setup to Flink 1.9.0.
-
+your Flink setup to Flink 1.9.0.</p>
-## List of Contributors
+<h2 id="list-of-contributors">List of Contributors</h2>
-We would like to thank all contributors who have made this release possible:
+<p>We would like to thank all contributors who have made this release possible:</p>
-Abdul Qadeer (abqadeer), Aitozi, Alberto Romero, Aleksey Pak, Alexander
+<p>Abdul Qadeer (abqadeer), Aitozi, Alberto Romero, Aleksey Pak, Alexander
Fedulov, Alice Yan, Aljoscha Krettek, Aloys, Andrew Duffy, Andrey Zagrebin,
Ankur, Artsem Semianenka, Benchao Li, Biao Liu, Bo WANG, Bowen L, Chesnay
Schepler, Clark Yang, Congxian Qiu, Cristian, Danny Chan, David Moravek, Dawid
@@ -3160,9 +3219,9 @@ shenlang.sl, shuai-xu, sunhaibotb, tianchen, tianchen92,
tison, tom_gong, vinoyang, vthinkxie, wanggeng3, wenhuitang, winifredtamg,
xl38154, xuyang1706, yangfei5, yanghua, yuzhao.cyz,
zhangxin516, zhangxinxing, zhaofaxian, zhijiang, zjuwangg, 林小铂,
-黄培松, 时无两丶.
+黄培松, 时无两丶.</p>
</description>
-<pubDate>Thu, 22 Aug 2019 02:30:00 +0000</pubDate>
+<pubDate>Thu, 22 Aug 2019 04:30:00 +0200</pubDate>
<link>https://flink.apache.org/news/2019/08/22/release-1.9.0.html</link>
<guid isPermaLink="true">/news/2019/08/22/release-1.9.0.html</guid>
</item>
@@ -3179,109 +3238,147 @@ zhangxin516, zhangxinxing, zhaofaxian, zhijiang, zjuwangg, 林小铂,
.tg .tg-center{text-align:center;vertical-align:center}
</style>
-In a [previous blog post]({{ site.baseurl }}/2019/06/05/flink-network-stack.html), we presented how Flink’s network stack works from the high-level abstractions to the low-level details. This second blog post in the series of network stack posts extends on this knowledge and discusses monitoring network-related metrics to identify effects such as backpressure or bottlenecks in throughput and latency. Although this post briefly covers what to do with backpressure, the topic of tuning the [...]
-
-{% toc %}
-
-## Monitoring
-
-Probably the most important part of network monitoring is [monitoring backpressure]({{ site.DOCS_BASE_URL }}flink-docs-release-1.8/monitoring/back_pressure.html), a situation where a system is receiving data at a higher rate than it can process¹. Such behaviour will result in the sender being backpressured and may be caused by two things:
+<p>In a <a href="/2019/06/05/flink-network-stack.html">previous blog post</a>, we presented how Flink’s network stack works from the high-level abstractions to the low-level details. This second blog post in the series of network stack posts extends on this knowledge and discusses monitoring network-related metrics to identify effects such as backpressure or bottlenecks in throughput and latency. Although this post briefly covers what to do with backpressure, [...]
+
+<div class="page-toc">
+<ul id="markdown-toc">
+ <li><a href="#monitoring" id="markdown-toc-monitoring">Monitoring</a> <ul>
+ <li><a href="#backpressure-monitor" id="markdown-toc-backpressure-monitor">Backpressure Monitor</a></li>
+ </ul>
+ </li>
+ <li><a href="#network-metrics" id="markdown-toc-network-metrics">Network Metrics</a> <ul>
+ <li><a href="#backpressure" id="markdown-toc-backpressure">Backpressure</a></li>
+ <li><a href="#resource-usage--throughput" id="markdown-toc-resource-usage--throughput">Resource Usage / Throughput</a></li>
+ <li><a href="#latency-tracking" id="markdown-toc-latency-tracking">Latency Tracking</a></li>
+ </ul>
+ </li>
+ <li><a href="#conclusion" id="markdown-toc-conclusion">Conclusion</a></li>
+</ul>
-* The receiver is slow.<br>
- This can happen because the receiver is backpressured itself, is unable to keep processing at the same rate as the sender, or is temporarily blocked by garbage collection, lack of system resources, or I/O.
+</div>
- * The network channel is slow.<br>
- Even though in such case the receiver is not (directly) involved, we call the sender backpressured due to a potential oversubscription on network bandwidth shared by all subtasks running on the same machine. Beware that, in addition to Flink’s network stack, there may be more network users, such as sources and sinks, distributed file systems (checkpointing, network-attached storage), logging, and metrics. A previous [capacity planning blog post](https://www.ververica.com/blog/how-to-si [...]
+<h2 id="monitoring">Monitoring</h2>
-<sup>1</sup> In case you are unfamiliar with backpressure and how it interacts with Flink, we recommend reading through [this blog post on backpressure](https://www.ververica.com/blog/how-flink-handles-backpressure) from 2015.
+<p>Probably the most important part of network monitoring is <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/monitoring/back_pressure.html">monitoring backpressure</a>, a situation where a system is receiving data at a higher rate than it can process¹. Such behaviour will result in the sender being backpressured and may be caused by two things:</p>
+<ul>
+ <li>
+ <p>The receiver is slow.<br />
+This can happen because the receiver is backpressured itself, is unable to keep processing at the same rate as the sender, or is temporarily blocked by garbage collection, lack of system resources, or I/O.</p>
+ </li>
+ <li>
+ <p>The network channel is slow.<br />
+ Even though in such case the receiver is not (directly) involved, we call the sender backpressured due to a potential oversubscription on network bandwidth shared by all subtasks running on the same machine. Beware that, in addition to Flink’s network stack, there may be more network users, such as sources and sinks, distributed file systems (checkpointing, network-attached storage), logging, and metrics. A previous <a href="https://www.ververica.com/blog/how-to-size-your-apach [...]
+ </li>
+</ul>
-<br>
-If backpressure occurs, it will bubble upstream and eventually reach your sources and slow them down. This is not a bad thing per-se and merely states that you lack resources for the current load. However, you may want to improve your job so that it can cope with higher loads without using more resources. In order to do so, you need to find (1) where (at which task/operator) the bottleneck is and (2) what is causing it. Flink offers two mechanisms for identifying where the bottleneck is:
+<p><sup>1</sup> In case you are unfamiliar with backpressure and how it interacts with Flink, we recommend reading through <a href="https://www.ververica.com/blog/how-flink-handles-backpressure">this blog post on backpressure</a> from 2015.</p>
- * directly via Flink’s web UI and its backpressure monitor, or
- * indirectly through some of the network metrics.
+<p><br />
+If backpressure occurs, it will bubble upstream and eventually reach your sources and slow them down. This is not a bad thing per-se and merely states that you lack resources for the current load. However, you may want to improve your job so that it can cope with higher loads without using more resources. In order to do so, you need to find (1) where (at which task/operator) the bottleneck is and (2) what is causing it. Flink offers two mechanisms for identifying where the bottleneck is: [...]
-Flink’s web UI is likely the first entry point for a quick troubleshooting but has some disadvantages that we will explain below. On the other hand, Flink’s network metrics are better suited for continuous monitoring and reasoning about the exact nature of the bottleneck causing backpressure. We will cover both in the sections below. In both cases, you need to identify the origin of backpressure from the sources to the sinks. Your starting point for the current and future investigations [...]
+<ul>
+ <li>directly via Flink’s web UI and its backpressure monitor, or</li>
+ <li>indirectly through some of the network metrics.</li>
+</ul>
+<p>Flink’s web UI is likely the first entry point for a quick troubleshooting but has some disadvantages that we will explain below. On the other hand, Flink’s network metrics are better suited for continuous monitoring and reasoning about the exact nature of the bottleneck causing backpressure. We will cover both in the sections below. In both cases, you need to identify the origin of backpressure from the sources to the sinks. Your starting point for the current and future invest [...]
-### Backpressure Monitor
+<h3 id="backpressure-monitor">Backpressure Monitor</h3>
-The [backpressure monitor]({{ site.DOCS_BASE_URL }}flink-docs-release-1.8/monitoring/back_pressure.html) is only exposed via Flink’s web UI². Since it's an active component that is only triggered on request, it is currently not available via metrics. The backpressure monitor samples the running tasks' threads on all TaskManagers via `Thread.getStackTrace()` and computes the number of samples where tasks were blocked on a buffer request. These tasks were either unable to send netw [...]
+<p>The <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/monitoring/back_pressure.html">backpressure monitor</a> is only exposed via Flink’s web UI². Since it’s an active component that is only triggered on request, it is currently not available via metrics. The backpressure monitor samples the running tasks’ threads on all TaskManagers via <code>Thread.getStackTrace()</code> and computes the number of samples where tasks were bl [...]
-* <span style="color:green">OK</span> for `ratio ≤ 0.10`,
-* <span style="color:orange">LOW</span> for `0.10 < Ratio ≤ 0.5`, and
-* <span style="color:red">HIGH</span> for `0.5 < Ratio ≤ 1`.
+<ul>
+ <li><span style="color:green">OK</span> for <code>ratio ≤ 0.10</code>,</li>
+ <li><span style="color:orange">LOW</span> for <code>0.10 &lt; Ratio ≤ 0.5</code>, and</li>
+ <li><span style="color:red">HIGH</span> for <code>0.5 &lt; Ratio ≤ 1</code>.</li>
+</ul>
-Although you can tune things like the refresh-interval, the number of samples, or the delay between samples, normally, you would not need to touch these since the defaults already give good-enough results.
+<p>Although you can tune things like the refresh-interval, the number of samples, or the delay between samples, normally, you would not need to touch these since the defaults already give good-enough results.</p>
<center>
-<img src="{{ site.baseurl }}/img/blog/2019-07-23-network-stack-2/back_pressure_sampling_high.png" width="600px" alt="Backpressure sampling:high"/>
+<img src="/img/blog/2019-07-23-network-stack-2/back_pressure_sampling_high.png" width="600px" alt="Backpressure sampling:high" />
</center>
-<sup>2</sup> You may also access the backpressure monitor via the REST API: `/jobs/:jobid/vertices/:vertexid/backpressure`
-
-
-<br>
-The backpressure monitor can help you find where (at which task/operator) backpressure originates from. However, it does not support you in further reasoning about the causes of it. Additionally, for larger jobs or higher parallelism, the backpressure monitor becomes too crowded to use and may also take some time to gather all information from all TaskManagers. Please also note that sampling may affect your running job’s performance.
-
-## Network Metrics
+<p><sup>2</sup> You may also access the backpressure monitor via the REST API: <code>/jobs/:jobid/vertices/:vertexid/backpressure</code></p>
-[Network]({{ site.DOCS_BASE_URL }}flink-docs-release-1.8/monitoring/metrics.html#network) and [task I/O]({{ site.DOCS_BASE_URL }}flink-docs-release-1.8/monitoring/metrics.html#io) metrics are more lightweight than the backpressure monitor and are continuously published for each running job. We can leverage those and get even more insights, not only for backpressure monitoring. The most relevant metrics for users are:
+<p><br />
+The backpressure monitor can help you find where (at which task/operator) backpressure originates from. However, it does not support you in further reasoning about the causes of it. Additionally, for larger jobs or higher parallelism, the backpressure monitor becomes too crowded to use and may also take some time to gather all information from all TaskManagers. Please also note that sampling may affect your running job’s performance.</p>
+<h2 id="network-metrics">Network Metrics</h2>
-* **<span style="color:orange">up to Flink 1.8:</span>** `outPoolUsage`, `inPoolUsage`<br>
- An estimate on the ratio of buffers used vs. buffers available in the respective local buffer pools.
- While interpreting `inPoolUsage` in Flink 1.5 - 1.8 with credit-based flow control, please note that this only relates to floating buffers (exclusive buffers are not part of the pool).
+<p><a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/monitoring/metrics.html#network">Network</a> and <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/monitoring/metrics.html#io">task I/O</a> metrics are more lightweight than the backpressure monitor and are continuously published for each running job. We can leverage those and get even more insights, not only for backpressure monitoring. The most re [...]
-* **<span style="color:green">Flink 1.9 and above:</span>** `outPoolUsage`, `inPoolUsage`, `floatingBuffersUsage`, `exclusiveBuffersUsage`<br>
- An estimate on the ratio of buffers used vs. buffers available in the respective local buffer pools.
- Starting with Flink 1.9, `inPoolUsage` is the sum of `floatingBuffersUsage` and `exclusiveBuffersUsage`.
-
-* `numRecordsOut`, `numRecordsIn`<br>
- Each metric comes with two scopes: one scoped to the operator and one scoped to the subtask. For network monitoring, the subtask-scoped metric is relevant and shows the total number of records it has sent/received. You may need to further look into these figures to extract the number of records within a certain time span or use the equivalent `…PerSecond` metrics.
-
-* `numBytesOut`, `numBytesInLocal`, `numBytesInRemote`<br>
- The total number of bytes this subtask has emitted or read from a local/remote source. These are also available as meters via `…PerSecond` metrics.
-
-* `numBuffersOut`, `numBuffersInLocal`, `numBuffersInRemote`<br>
- Similar to `numBytes…` but counting the number of network buffers.
+<ul>
+ <li>
+ <p><strong><span style="color:orange">up to Flink 1.8:</span></strong> <code>outPoolUsage</code>, <code>inPoolUsage</code><br />
+An estimate on the ratio of buffers used vs. buffers available in the respective local buffer pools.
+While interpreting <code>inPoolUsage</code> in Flink 1.5 - 1.8 with credit-based flow control, please note that this only relates to floating buffers (exclusive buffers are not part of the pool).</p>
+ </li>
+ <li>
+ <p><strong><span style="color:green">Flink 1.9 and above:</span></strong> <code>outPoolUsage</code>, <code>inPoolUsage</code>, <code>floatingBuffersUsage</code>, <code>exclusiveBuffersUsage</code><br />
+An estimate on the ratio of buffers used vs. buffers available in the respective local buffer pools.
+Starting with Flink 1.9, <code>inPoolUsage</code> is the sum of <code>floatingBuffersUsage</code> and <code>exclusiveBuffersUsage</code>.</p>
+ </li>
+ <li>
+ <p><code>numRecordsOut</code>, <code>numRecordsIn</code><br />
+Each metric comes with two scopes: one scoped to the operator and one scoped to the subtask. For network monitoring, the subtask-scoped metric is relevant and shows the total number of records it has sent/received. You may need to further look into these figures to extract the number of records within a certain time span or use the equivalent <code>…PerSecond</code> metrics.</p>
+ </li>
+ <li>
+ <p><code>numBytesOut</code>, <code>numBytesInLocal</code>, <code>numBytesInRemote</code><br />
+The total number of bytes this subtask has emitted or read from a local/remote source. These are also available as meters via <code>…PerSecond</code> metrics.</p>
+ </li>
+ <li>
+ <p><code>numBuffersOut</code>, <code>numBuffersInLocal</code>, <code>numBuffersInRemote</code><br />
+Similar to <code>numBytes…</code> but counting the number of network buffers.</p>
+ </li>
+</ul>
-<div class="alert alert-warning" markdown="1">
-<span class="label label-warning" style="display: inline-block"><span class="glyphicon glyphicon-warning-sign" aria-hidden="true"></span> Warning</span>
-For the sake of completeness and since they have been used in the past, we will briefly look at the `outputQueueLength` and `inputQueueLength` metrics. These are somewhat similar to the `[out,in]PoolUsage` metrics but show the number of buffers sitting in a sender subtask’s output queues and in a receiver subtask’s input queues, respectively. Reasoning about absolute numbers of buffers, however, is difficult and there is also a special subtlety with local channels: since a local input ch [...]
+<div class="alert alert-warning">
+ <p><span class="label label-warning" style="display: inline-block"><span class="glyphicon glyphicon-warning-sign" aria-hidden="true"></span> Warning</span>
+For the sake of completeness and since they have been used in the past, we will briefly look at the <code>outputQueueLength</code> and <code>inputQueueLength</code> metrics. These are somewhat similar to the <code>[out,in]PoolUsage</code> metrics but show the number of buffers sitting in a sender subtask’s output queues and in a receiver subtask’s input queues, respectively. Reasoning about absolute numbers of buffers, however, is difficult and there i [...]
-Overall, **we discourage the use of** `outputQueueLength` **and** `inputQueueLength` because their interpretation highly depends on the current parallelism of the operator and the configured numbers of exclusive and floating buffers. Instead, we recommend using the various `*PoolUsage` metrics which even reveal more detailed insight.
+ <p>Overall, <strong>we discourage the use of</strong> <code>outputQueueLength</code> <strong>and</strong> <code>inputQueueLength</code> because their interpretation highly depends on the current parallelism of the operator and the configured numbers of exclusive and floating buffers. Instead, we recommend using the various <code>*PoolUsage</code> metrics which even reveal more detailed insight.</p>
</div>
+<div class="alert alert-info">
+ <p><span class="label label-info" style="display: inline-block"><span class="glyphicon glyphicon-info-sign" aria-hidden="true"></span> Note</span>
+ If you reason about buffer usage, please keep the following in mind:</p>
-<div class="alert alert-info" markdown="1">
-<span class="label label-info" style="display: inline-block"><span class="glyphicon glyphicon-info-sign" aria-hidden="true"></span> Note</span>
- If you reason about buffer usage, please keep the following in mind:
-
-* Any outgoing channel which has been used at least once will always occupy one buffer (since Flink 1.5).
- * **<span style="color:orange">up to Flink 1.8:</span>** This buffer (even if empty!) was always counted as a backlog of 1 and thus receivers tried to reserve a floating buffer for it.
- * **<span style="color:green">Flink 1.9 and above:</span>** A buffer is only counted in the backlog if it is ready for consumption, i.e. it is full or was flushed (see FLINK-11082)
-* The receiver will only release a received buffer after deserialising the last record in it.
+ <ul>
+ <li>Any outgoing channel which has been used at least once will always occupy one buffer (since Flink 1.5).
+ <ul>
+ <li><strong><span style="color:orange">up to Flink 1.8:</span></strong> This buffer (even if empty!) was always counted as a backlog of 1 and thus receivers tried to reserve a floating buffer for it.</li>
+ <li><strong><span style="color:green">Flink 1.9 and above:</span></strong> A buffer is only counted in the backlog if it is ready for consumption, i.e. it is full or was flushed (see FLINK-11082)</li>
+ </ul>
+ </li>
+ <li>The receiver will only release a received buffer after deserialising the last record in it.</li>
+ </ul>
</div>
-The following sections make use of and combine these metrics to reason about backpressure and resource usage / efficiency with respect to throughput. A separate section will detail latency related metrics.
-
-
-### Backpressure
-
-Backpressure may be indicated by two different sets of metrics: (local) buffer pool usages as well as input/output queue lengths. They provide a different level of granularity but, unfortunately, none of these are exhaustive and there is room for interpretation. Because of the inherent problems with interpreting these queue lengths we will focus on the usage of input and output pools below which also provides more detail.
+<p>The following sections make use of and combine these metrics to reason about backpressure and resource usage / efficiency with respect to throughput. A separate section will detail latency related metrics.</p>
-* **If a subtask’s** `outPoolUsage` **is 100%**, it is backpressured. Whether the subtask is already blocking or still writing records into network buffers depends on how full the buffers are, that the `RecordWriters` are currently writing into.<br>
-<span class="glyphicon glyphicon-warning-sign" aria-hidden="true" style="color:orange;"></span> This is different to what the backpressure monitor is showing!
+<h3 id="backpressure">Backpressure</h3>
-* An `inPoolUsage` of 100% means that all floating buffers are assigned to channels and eventually backpressure will be exercised upstream. These floating buffers are in either of the following conditions: they are reserved for future use on a channel due to an exclusive buffer being utilised (remote input channels always try to maintain `#exclusive buffers` credits), they are reserved for a sender’s backlog and wait for data, they may contain data and are enqueued in an input channel, o [...]
+<p>Backpressure may be indicated by two different sets of metrics: (local) buffer pool usages as well as input/output queue lengths. They provide a different level of granularity but, unfortunately, none of these are exhaustive and there is room for interpretation. Because of the inherent problems with interpreting these queue lengths we will focus on the usage of input and output pools below which also provides more detail.</p>
-* **<span style="color:orange">up to Flink 1.8:</span>** Due to [FLINK-11082](https://issues.apache.org/jira/browse/FLINK-11082), an `inPoolUsage` of 100% is quite common even in normal situations.
-
-* **<span style="color:green">Flink 1.9 and above:</span>** If `inPoolUsage` is constantly around 100%, this is a strong indicator for exercising backpressure upstream.
+<ul>
+ <li>
+ <p><strong>If a subtask’s</strong> <code>outPoolUsage</code> <strong>is 100%</strong>, it is backpressured. Whether the subtask is already blocking or still writing records into network buffers depends on how full the buffers are, that the <code>RecordWriters</code> are currently writing into.<br />
+<span class="glyphicon glyphicon-warning-sign" aria-hidden="true" style="color:orange;"></span> This is different to what the backpressure monitor is showing!</p>
+ </li>
+ <li>
+ <p>An <code>inPoolUsage</code> of 100% means that all floating buffers are assigned to channels and eventually backpressure will be exercised upstream. These floating buffers are in either of the following conditions: they are reserved for future use on a channel due to an exclusive buffer being utilised (remote input channels always try to maintain <code>#exclusive buffers</code> credits), they are reserved for a sender’s backlog and wait for data, they [...]
+ </li>
+ <li>
+ <p><strong><span style="color:orange">up to Flink 1.8:</span></strong> Due to <a href="https://issues.apache.org/jira/browse/FLINK-11082">FLINK-11082</a>, an <code>inPoolUsage</code> of 100% is quite common even in normal situations.</p>
+ </li>
+ <li>
+ <p><strong><span style="color:green">Flink 1.9 and above:</span></strong> If <code>inPoolUsage</code> is constantly around 100%, this is a strong indicator for exercising backpressure upstream.</p>
+ </li>
+</ul>
-The following table summarises all combinations and their interpretation. Bear in mind, though, that backpressure may be minor or temporary (no need to look into it), on particular channels only, or caused by other JVM processes on a particular TaskManager, such as GC, synchronisation, I/O, resource shortage, instead of a specific subtask.
+<p>The following table summarises all combinations and their interpretation. Bear in mind, though, that backpressure may be minor or temporary (no need to look into it), on particular channels only, or caused by other JVM processes on a particular TaskManager, such as GC, synchronisation, I/O, resource shortage, instead of a specific subtask.</p>
<center>
<table class="tg">
@@ -3295,46 +3392,52 @@ The following table summarises all combinations and their interpretation. Bear i
<td class="tg-topcenter">
<span class="glyphicon glyphicon-ok-sign" aria-hidden="true" style="color:green;font-size:1.5em;"></span></td>
<td class="tg-topcenter">
- <span class="glyphicon glyphicon-warning-sign" aria-hidden="true" style="color:orange;font-size:1.5em;"></span><br>
+ <span class="glyphicon glyphicon-warning-sign" aria-hidden="true" style="color:orange;font-size:1.5em;"></span><br />
(backpressured, temporary situation: upstream is not backpressured yet or not anymore)</td>
</tr>
<tr>
<th class="tg-top" rowspan="2">
- <code>inPoolUsage</code> high<br>
+ <code>inPoolUsage</code> high<br />
(<strong><span style="color:green">Flink 1.9+</span></strong>)</th>
<td class="tg-topcenter">
- if all upstream tasks’<code>outPoolUsage</code> are low: <span class="glyphicon glyphicon-warning-sign" aria-hidden="true" style="color:orange;font-size:1.5em;"></span><br>
+ if all upstream tasks’<code>outPoolUsage</code> are low: <span class="glyphicon glyphicon-warning-sign" aria-hidden="true" style="color:orange;font-size:1.5em;"></span><br />
(may eventually cause backpressure)</td>
<td class="tg-topcenter" rowspan="2">
- <span class="glyphicon glyphicon-remove-sign" aria-hidden="true" style="color:red;font-size:1.5em;"></span><br>
+ <span class="glyphicon glyphicon-remove-sign" aria-hidden="true" style="color:red;font-size:1.5em;"></span><br />
(backpressured by downstream task(s) or network, probably forwarding backpressure upstream)</td>
</tr>
<tr>
- <td class="tg-topcenter">if any upstream task’s<code>outPoolUsage</code> is high: <span class="glyphicon glyphicon-remove-sign" aria-hidden="true" style="color:red;font-size:1.5em;"></span><br>
+ <td class="tg-topcenter">if any upstream task’s<code>outPoolUsage</code> is high: <span class="glyphicon glyphicon-remove-sign" aria-hidden="true" style="color:red;font-size:1.5em;"></span><br />
(may exercise backpressure upstream and may be the source of backpressure)</td>
</tr>
</table>
</center>
-<br>
-We may even reason more about the cause of backpressure by looking at the network metrics of the subtasks of two consecutive tasks:
+<p><br />
+We may even reason more about the cause of backpressure by looking at the network metrics of the subtasks of two consecutive tasks:</p>
-* If all subtasks of the receiver task have low `inPoolUsage` values and any upstream subtask’s `outPoolUsage` is high, then there may be a network bottleneck causing backpressure.
-Since network is a shared resource among all subtasks of a TaskManager, this may not directly originate from this subtask, but rather from various concurrent operations, e.g. checkpoints, other streams, external connections, or other TaskManagers/processes on the same machine.
+<ul>
+ <li>If all subtasks of the receiver task have low <code>inPoolUsage</code> values and any upstream subtask’s <code>outPoolUsage</code> is high, then there may be a network bottleneck causing backpressure.
+Since network is a shared resource among all subtasks of a TaskManager, this may not directly originate from this subtask, but rather from various concurrent operations, e.g. checkpoints, other streams, external connections, or other TaskManagers/processes on the same machine.</li>
+</ul>
-Backpressure can also be caused by all parallel instances of a task or by a single task instance. The first usually happens because the task is performing some time consuming operation that applies to all input partitions. The latter is usually the result of some kind of skew, either data skew or resource availability/allocation skew. In either case, you can find some hints on how to handle such situations in the [What to do with backpressure?](#span-classlabel-label-info-styledisplay-in [...]
+<p>Backpressure can also be caused by all parallel instances of a task or by a single task instance. The first usually happens because the task is performing some time consuming operation that applies to all input partitions. The latter is usually the result of some kind of skew, either data skew or resource availability/allocation skew. In either case, you can find some hints on how to handle such situations in the <a href="#span-classlabel-label-info-styledisplay-inline-b [...]
-<div class="alert alert-info" markdown="1">
-### <span class="glyphicon glyphicon-info-sign" aria-hidden="true"></span> Flink 1.9 and above
-{:.no_toc}
+<div class="alert alert-info">
+ <h3 class="no_toc" id="span-classglyphicon-glyphicon-info-sign-aria-hiddentruespan-flink-19-and-above"><span class="glyphicon glyphicon-info-sign" aria-hidden="true"></span> Flink 1.9 and above</h3>
-* If `floatingBuffersUsage` is not 100%, it is unlikely that there is backpressure. If it is 100% and any upstream task is backpressured, it suggests that this input is exercising backpressure on either a single, some or all input channels. To differentiate between those three situations you can use `exclusiveBuffersUsage`:
- * Assuming that `floatingBuffersUsage` is around 100%, the higher the `exclusiveBuffersUsage` the more input channels are backpressured. In an extreme case of `exclusiveBuffersUsage` being close to 100%, it means that all channels are backpressured.
+ <ul>
+ <li>If <code>floatingBuffersUsage</code> is not 100%, it is unlikely that there is backpressure. If it is 100% and any upstream task is backpressured, it suggests that this input is exercising backpressure on either a single, some or all input channels. To differentiate between those three situations you can use <code>exclusiveBuffersUsage</code>:
+ <ul>
+ <li>Assuming that <code>floatingBuffersUsage</code> is around 100%, the higher the <code>exclusiveBuffersUsage</code> the more input channels are backpressured. In an extreme case of <code>exclusiveBuffersUsage</code> being close to 100%, it means that all channels are backpressured.</li>
+ </ul>
+ </li>
+ </ul>
-<br>
-The relation between `exclusiveBuffersUsage`, `floatingBuffersUsage`, and the upstream tasks' `outPoolUsage` is summarised in the following table and extends on the table above with `inPoolUsage = floatingBuffersUsage + exclusiveBuffersUsage`:
+ <p><br />
+The relation between <code>exclusiveBuffersUsage</code>, <code>floatingBuffersUsage</code>, and the upstream tasks’ <code>outPoolUsage</code> is summarised in the following table and extends on the table above with <code>inPoolUsage = floatingBuffersUsage + exclusiveBuffersUsage</code>:</p>
-<center>
+ <center>
<table class="tg">
<tr>
<th></th>
@@ -3343,490 +3446,493 @@ The relation between `exclusiveBuffersUsage`, `floatingBuffersUsage`, and the up
</tr>
<tr>
<th class="tg-top" style="min-width:33%;">
- <code>floatingBuffersUsage</code> low +<br>
+ <code>floatingBuffersUsage</code> low +<br />
<em>all</em> upstream <code>outPoolUsage</code> low</th>
<td class="tg-center"><span class="glyphicon glyphicon-ok-sign" aria-hidden="true" style="color:green;font-size:1.5em;"></span></td>
<td class="tg-center">-<sup>3</sup></td>
</tr>
<tr>
<th class="tg-top" style="min-width:33%;">
- <code>floatingBuffersUsage</code> low +<br>
+ <code>floatingBuffersUsage</code> low +<br />
<em>any</em> upstream <code>outPoolUsage</code> high</th>
<td class="tg-center">
- <span class="glyphicon glyphicon-remove-sign" aria-hidden="true" style="color:red;font-size:1.5em;"></span><br>
+ <span class="glyphicon glyphicon-remove-sign" aria-hidden="true" style="color:red;font-size:1.5em;"></span><br />
(potential network bottleneck)</td>
<td class="tg-center">-<sup>3</sup></td>
</tr>
<tr>
<th class="tg-top" style="min-width:33%;">
- <code>floatingBuffersUsage</code> high +<br>
+ <code>floatingBuffersUsage</code> high +<br />
<em>all</em> upstream <code>outPoolUsage</code> low</th>
<td class="tg-center">
- <span class="glyphicon glyphicon-warning-sign" aria-hidden="true" style="color:orange;font-size:1.5em;"></span><br>
+ <span class="glyphicon glyphicon-warning-sign" aria-hidden="true" style="color:orange;font-size:1.5em;"></span><br />
(backpressure eventually appears on only some of the input channels)</td>
<td class="tg-center">
- <span class="glyphicon glyphicon-warning-sign" aria-hidden="true" style="color:orange;font-size:1.5em;"></span><br>
+ <span class="glyphicon glyphicon-warning-sign" aria-hidden="true" style="color:orange;font-size:1.5em;"></span><br />
(backpressure eventually appears on most or all of the input channels)</td>
</tr>
<tr>
<th class="tg-top" style="min-width:33%;">
- <code>floatingBuffersUsage</code> high +<br>
+ <code>floatingBuffersUsage</code> high +<br />
any upstream <code>outPoolUsage</code> high</th>
<td class="tg-center">
- <span class="glyphicon glyphicon-remove-sign" aria-hidden="true" style="color:red;font-size:1.5em;"></span><br>
+ <span class="glyphicon glyphicon-remove-sign" aria-hidden="true" style="color:red;font-size:1.5em;"></span><br />
(backpressure on only some of the input channels)</td>
<td class="tg-center">
- <span class="glyphicon glyphicon-remove-sign" aria-hidden="true" style="color:red;font-size:1.5em;"></span><br>
+ <span class="glyphicon glyphicon-remove-sign" aria-hidden="true" style="color:red;font-size:1.5em;"></span><br />
(backpressure on most or all of the input channels)</td>
</tr>
</table>
</center>
-<sup>3</sup> this should not happen
+ <p><sup>3</sup> this should not happen</p>
</div>
+<h3 id="resource-usage--throughput">Resource Usage / Throughput</h3>
-### Resource Usage / Throughput
+<p>Besides the obvious use of each individual metric mentioned above, there are also a few combinations providing useful insight into what is happening in the network stack:</p>
-Besides the obvious use of each individual metric mentioned above, there are also a few combinations providing useful insight into what is happening in the network stack:
-
-* Low throughput with frequent `outPoolUsage` values around 100% but low `inPoolUsage` on all receivers is an indicator that the round-trip-time of our credit-notification (depends on your network’s latency) is too high for the default number of exclusive buffers to make use of your bandwidth. Consider increasing the [buffers-per-channel]({{ site.DOCS_BASE_URL }}flink-docs-release-1.8/ops/config.html#taskmanager-network-memory-buffers-per-channel) parameter or try disabling credit-based [...]
-
-* Combining `numRecordsOut` and `numBytesOut` helps identifying average serialised record sizes which supports you in capacity planning for peak scenarios.
-
-* If you want to reason about buffer fill rates and the influence of the output flusher, you may combine `numBytesInRemote` with `numBuffersInRemote`. When tuning for throughput (and not latency!), low buffer fill rates may indicate reduced network efficiency. In such cases, consider increasing the buffer timeout.
-Please note that, as of Flink 1.8 and 1.9, `numBuffersOut` only increases for buffers getting full or for an event cutting off a buffer (e.g. a checkpoint barrier) and may lag behind. Please also note that reasoning about buffer fill rates on local channels is unnecessary since buffering is an optimisation technique for remote channels with limited effect on local channels.
-
-* You may also separate local from remote traffic using numBytesInLocal and numBytesInRemote but in most cases this is unnecessary.
+<ul>
+ <li>
+ <p>Low throughput with frequent <code>outPoolUsage</code> values around 100% but low <code>inPoolUsage</code> on all receivers is an indicator that the round-trip-time of our credit-notification (depends on your network’s latency) is too high for the default number of exclusive buffers to make use of your bandwidth. Consider increasing the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/ops/config.html#taskmanager-network-mem [...]
+ </li>
+ <li>
+ <p>Combining <code>numRecordsOut</code> and <code>numBytesOut</code> helps identifying average serialised record sizes which supports you in capacity planning for peak scenarios.</p>
+ </li>
+ <li>
+ <p>If you want to reason about buffer fill rates and the influence of the output flusher, you may combine <code>numBytesInRemote</code> with <code>numBuffersInRemote</code>. When tuning for throughput (and not latency!), low buffer fill rates may indicate reduced network efficiency. In such cases, consider increasing the buffer timeout.
+Please note that, as of Flink 1.8 and 1.9, <code>numBuffersOut</code> only increases for buffers getting full or for an event cutting off a buffer (e.g. a checkpoint barrier) and may lag behind. Please also note that reasoning about buffer fill rates on local channels is unnecessary since buffering is an optimisation technique for remote channels with limited effect on local channels.</p>
+ </li>
+ <li>
+ <p>You may also separate local from remote traffic using numBytesInLocal and numBytesInRemote but in most cases this is unnecessary.</p>
+ </li>
+</ul>
-<div class="alert alert-info" markdown="1">
-### <span class="glyphicon glyphicon-info-sign" aria-hidden="true"></span> What to do with Backpressure?
-{:.no_toc}
+<div class="alert alert-info">
+ <h3 class="no_toc" id="span-classglyphicon-glyphicon-info-sign-aria-hiddentruespan-what-to-do-with-backpressure"><span class="glyphicon glyphicon-info-sign" aria-hidden="true"></span> What to do with Backpressure?</h3>
-Assuming that you identified where the source of backpressure — a bottleneck — is located, the next step is to analyse why this is happening. Below, we list some potential causes of backpressure from the more basic to the more complex ones. We recommend to check the basic causes first, before diving deeper on the more complex ones and potentially drawing false conclusions.
+ <p>Assuming that you identified where the source of backpressure — a bottleneck — is located, the next step is to analyse why this is happening. Below, we list some potential causes of backpressure from the more basic to the more complex ones. We recommend to check the basic causes first, before diving deeper on the more complex ones and potentially drawing false conclusions.</p>
-Please also recall that backpressure might be temporary and the result of a load spike, checkpointing, or a job restart with a data backlog waiting to be processed. In that case, you can often just ignore it. Alternatively, keep in mind that the process of analysing and solving the issue can be affected by the intermittent nature of your bottleneck. Having said that, here are a couple of things to check.
+ <p>Please also recall that backpressure might be temporary and the result of a load spike, checkpointing, or a job restart with a data backlog waiting to be processed. In that case, you can often just ignore it. Alternatively, keep in mind that the process of analysing and solving the issue can be affected by the intermittent nature of your bottleneck. Having said that, here are a couple of things to check.</p>
-#### System Resources
+ <h4 id="system-resources">System Resources</h4>
-Firstly, you should check the incriminated machines’ basic resource usage like CPU, network, or disk I/O. If some resource is fully or heavily utilised you can do one of the following:
+ <p>Firstly, you should check the incriminated machines’ basic resource usage like CPU, network, or disk I/O. If some resource is fully or heavily utilised you can do one of the following:</p>
-1. Try to optimise your code. Code profilers are helpful in this case.
-2. Tune Flink for that specific resource.
-3. Scale out by increasing the parallelism and/or increasing the number of machines in the cluster.
+ <ol>
+ <li>Try to optimise your code. Code profilers are helpful in this case.</li>
+ <li>Tune Flink for that specific resource.</li>
+ <li>Scale out by increasing the parallelism and/or increasing the number of machines in the cluster.</li>
+ </ol>
-#### Garbage Collection
+ <h4 id="garbage-collection">Garbage Collection</h4>
-Oftentimes, performance issues arise from long GC pauses. You can verify whether you are in such a situation by either printing debug GC logs (via -`XX:+PrintGCDetails`) or by using some memory/GC profilers. Since dealing with GC issues is highly application-dependent and independent of Flink, we will not go into details here ([Oracle's Garbage Collection Tuning Guide](https://docs.oracle.com/javase/8/docs/technotes/guides/vm/gctuning/index.html) or [Plumbr’s Java Garbage Collection [...]
+ <p>Oftentimes, performance issues arise from long GC pauses. You can verify whether you are in such a situation by either printing debug GC logs (via -<code>XX:+PrintGCDetails</code>) or by using some memory/GC profilers. Since dealing with GC issues is highly application-dependent and independent of Flink, we will not go into details here (<a href="https://docs.oracle.com/javase/8/docs/technotes/guides/vm/gctuning/index.html">Oracle’s Garbage Collecti [...]
-#### CPU/Thread Bottleneck
+ <h4 id="cputhread-bottleneck">CPU/Thread Bottleneck</h4>
-Sometimes a CPU bottleneck might not be visible at first glance if one or a couple of threads are causing the CPU bottleneck while the CPU usage of the overall machine remains relatively low. For instance, a single CPU-bottlenecked thread on a 48-core machine would result in only 2% CPU use. Consider using code profilers for this as they can identify hot threads by showing each threads' CPU usage, for example.
+ <p>Sometimes a CPU bottleneck might not be visible at first glance if one or a couple of threads are causing the CPU bottleneck while the CPU usage of the overall machine remains relatively low. For instance, a single CPU-bottlenecked thread on a 48-core machine would result in only 2% CPU use. Consider using code profilers for this as they can identify hot threads by showing each threads’ CPU usage, for example.</p>
-#### Thread Contention
+ <h4 id="thread-contention">Thread Contention</h4>
-Similarly to the CPU/thread bottleneck issue above, a subtask may be bottlenecked due to high thread contention on shared resources. Again, CPU profilers are your best friend here! Consider looking for synchronisation overhead / lock contention in user code — although adding synchronisation in user code should be avoided and may even be dangerous! Also consider investigating shared system resources. The default JVM’s SSL implementation, for example, can become contented around the shared [...]
+ <p>Similarly to the CPU/thread bottleneck issue above, a subtask may be bottlenecked due to high thread contention on shared resources. Again, CPU profilers are your best friend here! Consider looking for synchronisation overhead / lock contention in user code — although adding synchronisation in user code should be avoided and may even be dangerous! Also consider investigating shared system resources. The default JVM’s SSL implementation, for example, can become contented around [...]
-#### Load Imbalance
+ <h4 id="load-imbalance">Load Imbalance</h4>
-If your bottleneck is caused by data skew, you can try to remove it or mitigate its impact by changing the data partitioning to separate heavy keys or by implementing local/pre-aggregation.
+ <p>If your bottleneck is caused by data skew, you can try to remove it or mitigate its impact by changing the data partitioning to separate heavy keys or by implementing local/pre-aggregation.</p>
-<br>
-This list is far from exhaustive. Generally, in order to reduce a bottleneck and thus backpressure, first analyse where it is happening and then find out why. The best place to start reasoning about the “why” is by checking what resources are fully utilised.
+ <p><br />
+This list is far from exhaustive. Generally, in order to reduce a bottleneck and thus backpressure, first analyse where it is happening and then find out why. The best place to start reasoning about the “why” is by checking what resources are fully utilised.</p>
</div>
-### Latency Tracking
+<h3 id="latency-tracking">Latency Tracking</h3>
-Tracking latencies at the various locations they may occur is a topic of its own. In this section, we will focus on the time records wait inside Flink’s network stack — including the system’s network connections. In low throughput scenarios, these latencies are influenced directly by the output flusher via the buffer timeout parameter or indirectly by any application code latencies. When processing a record takes longer than expected or when (multiple) timers fire at the same time — and [...]
+<p>Tracking latencies at the various locations they may occur is a topic of its own. In this section, we will focus on the time records wait inside Flink’s network stack — including the system’s network connections. In low throughput scenarios, these latencies are influenced directly by the output flusher via the buffer timeout parameter or indirectly by any application code latencies. When processing a record takes longer than expected or when (multiple) timers fire at the same ti [...]
-Flink offers some support for [tracking the latency]({{ site.DOCS_BASE_URL }}flink-docs-release-1.8/monitoring/metrics.html#latency-tracking) of records passing through the system (outside of user code). However, this is disabled by default (see below why!) and must be enabled by setting a latency tracking interval either in Flink’s [configuration via `metrics.latency.interval`]({{ site.DOCS_BASE_URL }}flink-docs-release-1.8/ops/config.html#metrics-latency-interval) or via [ExecutionConf [...]
+<p>Flink offers some support for <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/monitoring/metrics.html#latency-tracking">tracking the latency</a> of records passing through the system (outside of user code). However, this is disabled by default (see below why!) and must be enabled by setting a latency tracking interval either in Flink’s <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/ops/config.html#metrics-l [...]
-* `single`: one histogram for each operator subtask
-* `operator` (default): one histogram for each combination of source task and operator subtask
-* `subtask`: one histogram for each combination of source subtask and operator subtask (quadratic in the parallelism!)
+<ul>
+ <li><code>single</code>: one histogram for each operator subtask</li>
+ <li><code>operator</code> (default): one histogram for each combination of source task and operator subtask</li>
+ <li><code>subtask</code>: one histogram for each combination of source subtask and operator subtask (quadratic in the parallelism!)</li>
+</ul>
-These metrics are collected through special “latency markers”: each source subtask will periodically emit a special record containing the timestamp of its creation. The latency markers then flow alongside normal records while not overtaking them on the wire or inside a buffer queue. However, _a latency marker does not enter application logic_ and is overtaking records there. Latency markers therefore only measure the waiting time between the user code and not a full “end-to-end” latency. [...]
+<p>These metrics are collected through special “latency markers”: each source subtask will periodically emit a special record containing the timestamp of its creation. The latency markers then flow alongside normal records while not overtaking them on the wire or inside a buffer queue. However, <em>a latency marker does not enter application logic</em> and is overtaking records there. Latency markers therefore only measure the waiting time between the user code and not [...]
-Since `LatencyMarkers` sit in network buffers just like normal records, they will also wait for the buffer to be full or flushed due to buffer timeouts. When a channel is on high load, there is no added latency by the network buffering data. However, as soon as one channel is under low load, records and latency markers will experience an expected average delay of at most `buffer_timeout / 2`. This delay will add to each network connection towards a subtask and should be taken into accoun [...]
+<p>Since <code>LatencyMarkers</code> sit in network buffers just like normal records, they will also wait for the buffer to be full or flushed due to buffer timeouts. When a channel is on high load, there is no added latency by the network buffering data. However, as soon as one channel is under low load, records and latency markers will experience an expected average delay of at most <code>buffer_timeout / 2</code>. This delay will add to each network conne [...]
-By looking at the exposed latency tracking metrics for each subtask, for example at the 95th percentile, you should nevertheless be able to identify subtasks which are adding substantially to the overall source-to-sink latency and continue with optimising there.
+<p>By looking at the exposed latency tracking metrics for each subtask, for example at the 95th percentile, you should nevertheless be able to identify subtasks which are adding substantially to the overall source-to-sink latency and continue with optimising there.</p>
-<div class="alert alert-info" markdown="1">
-<span class="label label-info" style="display: inline-block"><span class="glyphicon glyphicon-info-sign" aria-hidden="true"></span> Note</span>
-Flink's latency markers assume that the clocks on all machines in the cluster are in sync. We recommend setting up an automated clock synchronisation service (like NTP) to avoid false latency results.
+<div class="alert alert-info">
+ <p><span class="label label-info" style="display: inline-block"><span class="glyphicon glyphicon-info-sign" aria-hidden="true"></span> Note</span>
+Flink’s latency markers assume that the clocks on all machines in the cluster are in sync. We recommend setting up an automated clock synchronisation service (like NTP) to avoid false latency results.</p>
</div>
-<div class="alert alert-warning" markdown="1">
-<span class="label label-warning" style="display: inline-block"><span class="glyphicon glyphicon-warning-sign" aria-hidden="true"></span> Warning</span>
-Enabling latency metrics can significantly impact the performance of the cluster (in particular for `subtask` granularity) due to the sheer amount of metrics being added as well as the use of histograms which are quite expensive to maintain. It is highly recommended to only use them for debugging purposes.
+<div class="alert alert-warning">
+ <p><span class="label label-warning" style="display: inline-block"><span class="glyphicon glyphicon-warning-sign" aria-hidden="true"></span> Warning</span>
+Enabling latency metrics can significantly impact the performance of the cluster (in particular for <code>subtask</code> granularity) due to the sheer amount of metrics being added as well as the use of histograms which are quite expensive to maintain. It is highly recommended to only use them for debugging purposes.</p>
</div>
+<h2 id="conclusion">Conclusion</h2>
-## Conclusion
-
-In the previous sections we discussed how to monitor Flink's network stack which primarily involves identifying backpressure: where it occurs, where it originates from, and (potentially) why it occurs. This can be executed in two ways: for simple cases and debugging sessions by using the backpressure monitor; for continuous monitoring, more in-depth analysis, and less runtime overhead by using Flink’s task and network stack metrics. Backpressure can be caused by the network layer its [...]
-
-Stay tuned for the third blog post in the series of network stack posts that will focus on tuning techniques and anti-patterns to avoid.
+<p>In the previous sections we discussed how to monitor Flink’s network stack which primarily involves identifying backpressure: where it occurs, where it originates from, and (potentially) why it occurs. This can be executed in two ways: for simple cases and debugging sessions by using the backpressure monitor; for continuous monitoring, more in-depth analysis, and less runtime overhead by using Flink’s task and network stack metrics. Backpressure can be caused by the network laye [...]
+<p>Stay tuned for the third blog post in the series of network stack posts that will focus on tuning techniques and anti-patterns to avoid.</p>
</description>
-<pubDate>Tue, 23 Jul 2019 15:30:00 +0000</pubDate>
+<pubDate>Tue, 23 Jul 2019 17:30:00 +0200</pubDate>
<link>https://flink.apache.org/2019/07/23/flink-network-stack-2.html</link>
<guid isPermaLink="true">/2019/07/23/flink-network-stack-2.html</guid>
</item>
<item>
<title>Apache Flink 1.8.1 Released</title>
-<description>The Apache Flink community released the first bugfix version of the Apache Flink 1.8 series.
-
-This release includes more than 40 fixes and minor improvements for Flink 1.8.1. The list below includes a detailed list of all improvements, sub-tasks and bug fixes.
-
-We highly recommend all users to upgrade to Flink 1.8.1.
-
-Updated Maven dependencies:
-
-```xml
-<dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-java</artifactId>
- <version>1.8.1</version>
-</dependency>
-<dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-streaming-java_2.11</artifactId>
- <version>1.8.1</version>
-</dependency>
-<dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-clients_2.11</artifactId>
- <version>1.8.1</version>
-</dependency>
-```
-
-You can find the binaries on the updated [Downloads page]({{ site.baseurl }}/downloads.html).
-
-List of resolved issues:
-
+<description><p>The Apache Flink community released the first bugfix version of the Apache Flink 1.8 series.</p>
+
+<p>This release includes more than 40 fixes and minor improvements for Flink 1.8.1. The list below includes a detailed list of all improvements, sub-tasks and bug fixes.</p>
+
+<p>We highly recommend all users to upgrade to Flink 1.8.1.</p>
+
+<p>Updated Maven dependencies:</p>
+
+<div class="highlight"><pre><code class="language-xml"><span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-java<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.8.1<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span>
+<span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-streaming-java_2.11<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.8.1<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span>
+<span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-clients_2.11<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.8.1<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span></code></pre></div>
+
+<p>You can find the binaries on the updated <a href="/downloads.html">Downloads page</a>.</p>
+
+<p>List of resolved issues:</p>
+
<h2> Sub-task
</h2>
<ul>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10921'>FLINK-10921</a>] - Prioritize shard consumers in Kinesis Consumer by event time
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10921">FLINK-10921</a>] - Prioritize shard consumers in Kinesis Consumer by event time
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-12617'>FLINK-12617</a>] - StandaloneJobClusterEntrypoint should default to random JobID for non-HA setups
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12617">FLINK-12617</a>] - StandaloneJobClusterEntrypoint should default to random JobID for non-HA setups
</li>
</ul>
-
+
<h2> Bug
</h2>
<ul>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-9445'>FLINK-9445</a>] - scala-shell uses plain java command
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-9445">FLINK-9445</a>] - scala-shell uses plain java command
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10455'>FLINK-10455</a>] - Potential Kafka producer leak in case of failures
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10455">FLINK-10455</a>] - Potential Kafka producer leak in case of failures
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10941'>FLINK-10941</a>] - Slots prematurely released which still contain unconsumed data
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10941">FLINK-10941</a>] - Slots prematurely released which still contain unconsumed data
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-11059'>FLINK-11059</a>] - JobMaster may continue using an invalid slot if releasing idle slot meet a timeout
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11059">FLINK-11059</a>] - JobMaster may continue using an invalid slot if releasing idle slot meet a timeout
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-11107'>FLINK-11107</a>] - Avoid memory stateBackend to create arbitrary folders under HA path when no checkpoint path configured
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11107">FLINK-11107</a>] - Avoid memory stateBackend to create arbitrary folders under HA path when no checkpoint path configured
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-11897'>FLINK-11897</a>] - ExecutionGraphSuspendTest does not wait for all tasks to be submitted
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11897">FLINK-11897</a>] - ExecutionGraphSuspendTest does not wait for all tasks to be submitted
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-11915'>FLINK-11915</a>] - DataInputViewStream skip returns wrong value
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11915">FLINK-11915</a>] - DataInputViewStream skip returns wrong value
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-11987'>FLINK-11987</a>] - Kafka producer occasionally throws NullpointerException
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11987">FLINK-11987</a>] - Kafka producer occasionally throws NullpointerException
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-12009'>FLINK-12009</a>] - Wrong check message about heartbeat interval for HeartbeatServices
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12009">FLINK-12009</a>] - Wrong check message about heartbeat interval for HeartbeatServices
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-12042'>FLINK-12042</a>] - RocksDBStateBackend mistakenly uses default filesystem
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12042">FLINK-12042</a>] - RocksDBStateBackend mistakenly uses default filesystem
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-12112'>FLINK-12112</a>] - AbstractTaskManagerProcessFailureRecoveryTest process output logging does not work properly
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12112">FLINK-12112</a>] - AbstractTaskManagerProcessFailureRecoveryTest process output logging does not work properly
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-12132'>FLINK-12132</a>] - The example in /docs/ops/deployment/yarn_setup.md should be updated due to the change FLINK-2021
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12132">FLINK-12132</a>] - The example in /docs/ops/deployment/yarn_setup.md should be updated due to the change FLINK-2021
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-12184'>FLINK-12184</a>] - HistoryServerArchiveFetcher isn&#39;t compatible with old version
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12184">FLINK-12184</a>] - HistoryServerArchiveFetcher isn&#39;t compatible with old version
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-12219'>FLINK-12219</a>] - Yarn application can&#39;t stop when flink job failed in per-job yarn cluster mode
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12219">FLINK-12219</a>] - Yarn application can&#39;t stop when flink job failed in per-job yarn cluster mode
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-12247'>FLINK-12247</a>] - fix NPE when writing an archive file to a FileSystem
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12247">FLINK-12247</a>] - fix NPE when writing an archive file to a FileSystem
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-12260'>FLINK-12260</a>] - Slot allocation failure by taskmanager registration timeout and race
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12260">FLINK-12260</a>] - Slot allocation failure by taskmanager registration timeout and race
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-12296'>FLINK-12296</a>] - Data loss silently in RocksDBStateBackend when more than one operator(has states) chained in a single task
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12296">FLINK-12296</a>] - Data loss silently in RocksDBStateBackend when more than one operator(has states) chained in a single task
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-12297'>FLINK-12297</a>] - Make ClosureCleaner recursive
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12297">FLINK-12297</a>] - Make ClosureCleaner recursive
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-12301'>FLINK-12301</a>] - Scala value classes inside case classes cannot be serialized anymore in Flink 1.8.0
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12301">FLINK-12301</a>] - Scala value classes inside case classes cannot be serialized anymore in Flink 1.8.0
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-12342'>FLINK-12342</a>] - Yarn Resource Manager Acquires Too Many Containers
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12342">FLINK-12342</a>] - Yarn Resource Manager Acquires Too Many Containers
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-12375'>FLINK-12375</a>] - flink-container job jar does not have read permissions
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12375">FLINK-12375</a>] - flink-container job jar does not have read permissions
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-12416'>FLINK-12416</a>] - Docker build script fails on symlink creation ln -s
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12416">FLINK-12416</a>] - Docker build script fails on symlink creation ln -s
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-12544'>FLINK-12544</a>] - Deadlock while releasing memory and requesting segment concurrent in SpillableSubpartition
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12544">FLINK-12544</a>] - Deadlock while releasing memory and requesting segment concurrent in SpillableSubpartition
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-12547'>FLINK-12547</a>] - Deadlock when the task thread downloads jars using BlobClient
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12547">FLINK-12547</a>] - Deadlock when the task thread downloads jars using BlobClient
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-12646'>FLINK-12646</a>] - Use reserved IP as unrouteable IP in RestClientTest
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12646">FLINK-12646</a>] - Use reserved IP as unrouteable IP in RestClientTest
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-12688'>FLINK-12688</a>] - Make serializer lazy initialization thread safe in StateDescriptor
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12688">FLINK-12688</a>] - Make serializer lazy initialization thread safe in StateDescriptor
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-12740'>FLINK-12740</a>] - SpillableSubpartitionTest deadlocks on Travis
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12740">FLINK-12740</a>] - SpillableSubpartitionTest deadlocks on Travis
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-12835'>FLINK-12835</a>] - Time conversion is wrong in ManualClock
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12835">FLINK-12835</a>] - Time conversion is wrong in ManualClock
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-12863'>FLINK-12863</a>] - Race condition between slot offerings and AllocatedSlotReport
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12863">FLINK-12863</a>] - Race condition between slot offerings and AllocatedSlotReport
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-12865'>FLINK-12865</a>] - State inconsistency between RM and TM on the slot status
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12865">FLINK-12865</a>] - State inconsistency between RM and TM on the slot status
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-12871'>FLINK-12871</a>] - Wrong SSL setup examples in docs
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12871">FLINK-12871</a>] - Wrong SSL setup examples in docs
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-12895'>FLINK-12895</a>] - TaskManagerProcessFailureBatchRecoveryITCase.testTaskManagerProcessFailure failed on travis
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12895">FLINK-12895</a>] - TaskManagerProcessFailureBatchRecoveryITCase.testTaskManagerProcessFailure failed on travis
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-12896'>FLINK-12896</a>] - TaskCheckpointStatisticDetailsHandler uses wrong value for JobID when archiving
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12896">FLINK-12896</a>] - TaskCheckpointStatisticDetailsHandler uses wrong value for JobID when archiving
</li>
</ul>
-
+
<h2> Improvement
</h2>
<ul>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-11126'>FLINK-11126</a>] - Filter out AMRMToken in the TaskManager credentials
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11126">FLINK-11126</a>] - Filter out AMRMToken in the TaskManager credentials
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-12137'>FLINK-12137</a>] - Add more proper explanation on flink streaming connectors
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12137">FLINK-12137</a>] - Add more proper explanation on flink streaming connectors
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-12169'>FLINK-12169</a>] - Improve Javadoc of MessageAcknowledgingSourceBase
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12169">FLINK-12169</a>] - Improve Javadoc of MessageAcknowledgingSourceBase
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-12378'>FLINK-12378</a>] - Consolidate FileSystem Documentation
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12378">FLINK-12378</a>] - Consolidate FileSystem Documentation
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-12391'>FLINK-12391</a>] - Add timeout to transfer.sh
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12391">FLINK-12391</a>] - Add timeout to transfer.sh
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-12539'>FLINK-12539</a>] - StreamingFileSink: Make the class extendable to customize for different usecases
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12539">FLINK-12539</a>] - StreamingFileSink: Make the class extendable to customize for different usecases
</li>
</ul>
-
+
<h2> Test
</h2>
<ul>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-12350'>FLINK-12350</a>] - RocksDBStateBackendTest doesn&#39;t cover the incremental checkpoint code path
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12350">FLINK-12350</a>] - RocksDBStateBackendTest doesn&#39;t cover the incremental checkpoint code path
</li>
</ul>
-
+
<h2> Task
</h2>
<ul>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-12460'>FLINK-12460</a>] - Change taskmanager.tmp.dirs to io.tmp.dirs in configuration docs
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12460">FLINK-12460</a>] - Change taskmanager.tmp.dirs to io.tmp.dirs in configuration docs
</li>
</ul>
- </description>
-<pubDate>Tue, 02 Jul 2019 12:00:00 +0000</pubDate>
+
+</description>
+<pubDate>Tue, 02 Jul 2019 14:00:00 +0200</pubDate>
<link>https://flink.apache.org/news/2019/07/02/release-1.8.1.html</link>
<guid isPermaLink="true">/news/2019/07/02/release-1.8.1.html</guid>
</item>
<item>
<title>A Practical Guide to Broadcast State in Apache Flink</title>
-<description>Since version 1.5.0, Apache Flink features a new type of state which is called Broadcast State. In this post, we explain what Broadcast State is, and show an example of how it can be applied to an application that evaluates dynamic patterns on an event stream. We walk you through the processing steps and the source code to implement this application in practice.
+<description><p>Since version 1.5.0, Apache Flink features a new type of state which is called Broadcast State. In this post, we explain what Broadcast State is, and show an example of how it can be applied to an application that evaluates dynamic patterns on an event stream. We walk you through the processing steps and the source code to implement this application in practice.</p>
-## What is Broadcast State?
+<h2 id="what-is-broadcast-state">What is Broadcast State?</h2>
-The Broadcast State can be used to combine and jointly process two streams of events in a specific way. The events of the first stream are broadcasted to all parallel instances of an operator, which maintains them as state. The events of the other stream are not broadcasted but sent to individual instances of the same operator and processed together with the events of the broadcasted stream.
-The new broadcast state is a natural fit for applications that need to join a low-throughput and a high-throughput stream or need to dynamically update their processing logic. We will use a concrete example of the latter use case to explain the broadcast state and show its API in more detail in the remainder of this post.
+<p>The Broadcast State can be used to combine and jointly process two streams of events in a specific way. The events of the first stream are broadcasted to all parallel instances of an operator, which maintains them as state. The events of the other stream are not broadcasted but sent to individual instances of the same operator and processed together with the events of the broadcasted stream.
+The new broadcast state is a natural fit for applications that need to join a low-throughput and a high-throughput stream or need to dynamically update their processing logic. We will use a concrete example of the latter use case to explain the broadcast state and show its API in more detail in the remainder of this post.</p>
-## Dynamic Pattern Evaluation with Broadcast State
+<h2 id="dynamic-pattern-evaluation-with-broadcast-state">Dynamic Pattern Evaluation with Broadcast State</h2>
-Imagine an e-commerce website that captures the interactions of all users as a stream of user actions. The company that operates the website is interested in analyzing the interactions to increase revenue, improve the user experience, and detect and prevent malicious behavior.
-The website implements a streaming application that detects a pattern on the stream of user events. However, the company wants to avoid modifying and redeploying the application every time the pattern changes. Instead, the application ingests a second stream of patterns and updates its active pattern when it receives a new pattern from the pattern stream. In the following, we discuss this application step-by-step and show how it leverages the broadcast state feature in Apache Flink.
+<p>Imagine an e-commerce website that captures the interactions of all users as a stream of user actions. The company that operates the website is interested in analyzing the interactions to increase revenue, improve the user experience, and detect and prevent malicious behavior.
+The website implements a streaming application that detects a pattern on the stream of user events. However, the company wants to avoid modifying and redeploying the application every time the pattern changes. Instead, the application ingests a second stream of patterns and updates its active pattern when it receives a new pattern from the pattern stream. In the following, we discuss this application step-by-step and show how it leverages the broadcast state feature in Apache Flink.</p>
<center>
-<img src="{{ site.baseurl }}/img/blog/broadcastState/fig1.png" width="600px" alt="Broadcast State in Apache Flink."/>
+<img src="/img/blog/broadcastState/fig1.png" width="600px" alt="Broadcast State in Apache Flink." />
</center>
-<br>
-
-Our example application ingests two data streams. The first stream provides user actions on the website and is illustrated on the top left side of the above figure. A user interaction event consists of the type of the action (user login, user logout, add to cart, or complete payment) and the id of the user, which is encoded by color. The user action event stream in our illustration contains a logout action of User 1001 followed by a payment-complete event for User 1003, and an “add-to-ca [...]
+<p><br /></p>
-The second stream provides action patterns that the application will evaluate. A pattern consists of two consecutive actions. In the figure above, the pattern stream contains the following two:
+<p>Our example application ingests two data streams. The first stream provides user actions on the website and is illustrated on the top left side of the above figure. A user interaction event consists of the type of the action (user login, user logout, add to cart, or complete payment) and the id of the user, which is encoded by color. The user action event stream in our illustration contains a logout action of User 1001 followed by a payment-complete event for User 1003, and an “ [...]
-* Pattern #1: A user logs in and immediately logs out without browsing additional pages on the e-commerce website.
-* Pattern #2: A user adds an item to the shopping cart and logs out without completing the purchase.
+<p>The second stream provides action patterns that the application will evaluate. A pattern consists of two consecutive actions. In the figure above, the pattern stream contains the following two:</p>
+<ul>
+ <li>Pattern #1: A user logs in and immediately logs out without browsing additional pages on the e-commerce website.</li>
+ <li>Pattern #2: A user adds an item to the shopping cart and logs out without completing the purchase.</li>
+</ul>
-Such patterns help a business in better analyzing user behavior, detecting malicious actions, and improving the website experience. For example, in the case of items being added to a shopping cart with no follow up purchase, the website team can take appropriate actions to understand better the reasons why users don’t complete a purchase and initiate specific programs to improve the website conversion (such as providing discount codes, limited free shipping offers etc.)
+<p>Such patterns help a business in better analyzing user behavior, detecting malicious actions, and improving the website experience. For example, in the case of items being added to a shopping cart with no follow up purchase, the website team can take appropriate actions to understand better the reasons why users don’t complete a purchase and initiate specific programs to improve the website conversion (such as providing discount codes, limited free shipping offers etc.)</p>
-On the right-hand side, the figure shows three parallel tasks of an operator that ingest the pattern and user action streams, evaluate the patterns on the action stream, and emit pattern matches downstream. For the sake of simplicity, the operator in our example only evaluates a single pattern with exactly two subsequent actions. The currently active pattern is replaced when a new pattern is received from the pattern stream. In principle, the operator could also be implemented to evaluat [...]
+<p>On the right-hand side, the figure shows three parallel tasks of an operator that ingest the pattern and user action streams, evaluate the patterns on the action stream, and emit pattern matches downstream. For the sake of simplicity, the operator in our example only evaluates a single pattern with exactly two subsequent actions. The currently active pattern is replaced when a new pattern is received from the pattern stream. In principle, the operator could also be implemented t [...]
-We will describe how the pattern matching application processes the user action and pattern streams.
+<p>We will describe how the pattern matching application processes the user action and pattern streams.</p>
<center>
-<img src="{{ site.baseurl }}/img/blog/broadcastState/fig2.png" width="600px" alt="Broadcast State in Apache Flink."/>
+<img src="/img/blog/broadcastState/fig2.png" width="600px" alt="Broadcast State in Apache Flink." />
</center>
-<br>
+<p><br /></p>
-First a pattern is sent to the operator. The pattern is broadcasted to all three parallel tasks of the operator. The tasks store the pattern in their broadcast state. Since the broadcast state should only be updated using broadcasted data, the state of all tasks is always expected to be the same.
+<p>First a pattern is sent to the operator. The pattern is broadcasted to all three parallel tasks of the operator. The tasks store the pattern in their broadcast state. Since the broadcast state should only be updated using broadcasted data, the state of all tasks is always expected to be the same.</p>
<center>
-<img src="{{ site.baseurl }}/img/blog/broadcastState/fig3.png" width="600px" alt="Broadcast State in Apache Flink."/>
+<img src="/img/blog/broadcastState/fig3.png" width="600px" alt="Broadcast State in Apache Flink." />
</center>
-<br>
+<p><br /></p>
-Next, the first user actions are partitioned on the user id and shipped to the operator tasks. The partitioning ensures that all actions of the same user are processed by the same task. The figure above shows the state of the application after the first pattern and the first three action events were consumed by the operator tasks.
+<p>Next, the first user actions are partitioned on the user id and shipped to the operator tasks. The partitioning ensures that all actions of the same user are processed by the same task. The figure above shows the state of the application after the first pattern and the first three action events were consumed by the operator tasks.</p>
-When a task receives a new user action, it evaluates the currently active pattern by looking at the user’s latest and previous actions. For each user, the operator stores the previous action in the keyed state. Since the tasks in the figure above only received a single action for each user so far (we just started the application), the pattern does not need to be evaluated. Finally, the previous action in the user’s keyed state is updated to the latest action, to be able to look it up whe [...]
+<p>When a task receives a new user action, it evaluates the currently active pattern by looking at the user’s latest and previous actions. For each user, the operator stores the previous action in the keyed state. Since the tasks in the figure above only received a single action for each user so far (we just started the application), the pattern does not need to be evaluated. Finally, the previous action in the user’s keyed state is updated to the latest action, to be able to look [...]
<center>
-<img src="{{ site.baseurl }}/img/blog/broadcastState/fig4.png" width="600px" alt="Broadcast State in Apache Flink."/>
+<img src="/img/blog/broadcastState/fig4.png" width="600px" alt="Broadcast State in Apache Flink." />
</center>
-<br>
+<p><br /></p>
-After the first three actions are processed, the next event, the logout action of User 1001, is shipped to the task that processes the events of User 1001. When the task receives the actions, it looks up the current pattern from the broadcast state and the previous action of User 1001. Since the pattern matches both actions, the task emits a pattern match event. Finally, the task updates its keyed state by overriding the previous event with the latest action.
+<p>After the first three actions are processed, the next event, the logout action of User 1001, is shipped to the task that processes the events of User 1001. When the task receives the actions, it looks up the current pattern from the broadcast state and the previous action of User 1001. Since the pattern matches both actions, the task emits a pattern match event. Finally, the task updates its keyed state by overriding the previous event with the latest action.</p>
<center>
-<img src="{{ site.baseurl }}/img/blog/broadcastState/fig5.png" width="600px" alt="Broadcast State in Apache Flink."/>
+<img src="/img/blog/broadcastState/fig5.png" width="600px" alt="Broadcast State in Apache Flink." />
</center>
-<br>
+<p><br /></p>
-When a new pattern arrives in the pattern stream, it is broadcasted to all tasks and each task updates its broadcast state by replacing the current pattern with the new one.
+<p>When a new pattern arrives in the pattern stream, it is broadcasted to all tasks and each task updates its broadcast state by replacing the current pattern with the new one.</p>
<center>
-<img src="{{ site.baseurl }}/img/blog/broadcastState/fig6.png" width="600px" alt="Broadcast State in Apache Flink."/>
+<img src="/img/blog/broadcastState/fig6.png" width="600px" alt="Broadcast State in Apache Flink." />
</center>
-<br>
-
-Once the broadcast state is updated with a new pattern, the matching logic continues as before, i.e., user action events are partitioned by key and evaluated by the responsible task.
-
+<p><br /></p>
-## How to Implement an Application with Broadcast State?
+<p>Once the broadcast state is updated with a new pattern, the matching logic continues as before, i.e., user action events are partitioned by key and evaluated by the responsible task.</p>
-Until now, we conceptually discussed the application and explained how it uses broadcast state to evaluate dynamic patterns over event streams. Next, we’ll show how to implement the example application with Flink’s DataStream API and the broadcast state feature.
+<h2 id="how-to-implement-an-application-with-broadcast-state">How to Implement an Application with Broadcast State?</h2>
-Let’s start with the input data of the application. We have two data streams, actions, and patterns. At this point, we don’t really care where the streams come from. The streams could be ingested from Apache Kafka or Kinesis or any other system. Action and Pattern are Pojos with two fields each:
+<p>Until now, we conceptually discussed the application and explained how it uses broadcast state to evaluate dynamic patterns over event streams. Next, we’ll show how to implement the example application with Flink’s DataStream API and the broadcast state feature.</p>
-```java
-DataStream<Action> actions = ???
-DataStream<Pattern> patterns = ???
-```
+<p>Let’s start with the input data of the application. We have two data streams, actions, and patterns. At this point, we don’t really care where the streams come from. The streams could be ingested from Apache Kafka or Kinesis or any other system. Action and Pattern are Pojos with two fields each:</p>
-`Action` and `Pattern` are Pojos with two fields each:
+<div class="highlight"><pre><code class="language-java"><span class="n">DataStream</span><span class="o">&lt;</span><span class="n">Action</span><span class="o">&gt;</span> <span class="n">actions</span> <span class="o">=</span> <span class="o">???</span>
+<span class="n">DataStream</span><span class="o">&lt;</span><span class="n">Pattern</span><span class="o">&gt;</span> <span class="n">patterns</span> <span class="o">=</span> <span class="o">???</span></code></pre></div>
-- `Action: Long userId, String action`
+<p><code>Action</code> and <code>Pattern</code> are Pojos with two fields each:</p>
-- `Pattern: String firstAction, String secondAction`
+<ul>
+ <li>
+ <p><code>Action: Long userId, String action</code></p>
+ </li>
+ <li>
+ <p><code>Pattern: String firstAction, String secondAction</code></p>
+ </li>
+</ul>
-As a first step, we key the action stream on the `userId` attribute.
+<p>As a first step, we key the action stream on the <code>userId</code> attribute.</p>
-```java
-KeyedStream<Action, Long> actionsByUser = actions
- .keyBy((KeySelector<Action, Long>) action -> action.userId);
-```
+<div class="highlight"><pre><code class="language-java"><span class="n">KeyedStream</span><span class="o">&lt;</span><span class="n">Action</span><span class="o">,</span> <span class="n">Long</span><span class="o">&gt;</span> <span class="n">actionsByUser</span> <span class="o"& [...]
+ <span class="o">.</span><span class="na">keyBy</span><span class="o">((</span><span class="n">KeySelector</span><span class="o">&lt;</span><span class="n">Action</span><span class="o">,</span> <span class="n">Long</span><span class="o">&gt;)</span> <span class="n">act [...]
-Next, we prepare the broadcast state. Broadcast state is always represented as `MapState`, the most versatile state primitive that Flink provides.
+<p>Next, we prepare the broadcast state. Broadcast state is always represented as <code>MapState</code>, the most versatile state primitive that Flink provides.</p>
-```java
-MapStateDescriptor<Void, Pattern> bcStateDescriptor =
- new MapStateDescriptor<>("patterns", Types.VOID, Types.POJO(Pattern.class));
-```
+<div class="highlight"><pre><code class="language-java"><span class="n">MapStateDescriptor</span><span class="o">&lt;</span><span class="n">Void</span><span class="o">,</span> <span class="n">Pattern</span><span class="o">&gt;</span> <span class="n">bcStateDescriptor</span> <span class=&q [...]
+ <span class="k">new</span> <span class="n">MapStateDescriptor</span><span class="o">&lt;&gt;(</span><span class="s">&quot;patterns&quot;</span><span class="o">,</span> <span class="n">Types</span><span class="o">.</span><span class="na">VOID</span><span class="o">,</span> < [...]
-Since our application only evaluates and stores a single `Pattern` at a time, we configure the broadcast state as a `MapState` with key type `Void` and value type `Pattern`. The `Pattern` is always stored in the `MapState` with `null` as key.
+<p>Since our application only evaluates and stores a single <code>Pattern</code> at a time, we configure the broadcast state as a <code>MapState</code> with key type <code>Void</code> and value type <code>Pattern</code>. The <code>Pattern</code> is always stored in the <code>MapState</code> with <code>null</code> as key.</p>
-```java
-BroadcastStream<Pattern> bcedPatterns = patterns.broadcast(bcStateDescriptor);
-```
-Using the `MapStateDescriptor` for the broadcast state, we apply the `broadcast()` transformation on the patterns stream and receive a `BroadcastStream bcedPatterns`.
+<div class="highlight"><pre><code class="language-java"><span class="n">BroadcastStream</span><span class="o">&lt;</span><span class="n">Pattern</span><span class="o">&gt;</span> <span class="n">bcedPatterns</span> <span class="o">=</span> <span class="n">patterns</span><span class=" [...]
+<p>Using the <code>MapStateDescriptor</code> for the broadcast state, we apply the <code>broadcast()</code> transformation on the patterns stream and receive a <code>BroadcastStream bcedPatterns</code>.</p>
-```java
-DataStream<Tuple2<Long, Pattern>> matches = actionsByUser
- .connect(bcedPatterns)
- .process(new PatternEvaluator());
-```
+<div class="highlight"><pre><code class="language-java"><span class="n">DataStream</span><span class="o">&lt;</span><span class="n">Tuple2</span><span class="o">&lt;</span><span class="n">Long</span><span class="o">,</span> <span class="n">Pattern</span><span class="o">&g [...]
+ <span class="o">.</span><span class="na">connect</span><span class="o">(</span><span class="n">bcedPatterns</span><span class="o">)</span>
+ <span class="o">.</span><span class="na">process</span><span class="o">(</span><span class="k">new</span> <span class="nf">PatternEvaluator</span><span class="o">());</span></code></pre></div>
-After we obtained the keyed `actionsByUser` stream and the broadcasted `bcedPatterns` stream, we `connect()` both streams and apply a `PatternEvaluator` on the connected streams. `PatternEvaluator` is a custom function that implements the `KeyedBroadcastProcessFunction` interface. It applies the pattern matching logic that we discussed before and emits `Tuple2<Long, Pattern>` records which contain the user id and the matched pattern.
+<p>After we obtained the keyed <code>actionsByUser</code> stream and the broadcasted <code>bcedPatterns</code> stream, we <code>connect()</code> both streams and apply a <code>PatternEvaluator</code> on the connected streams. <code>PatternEvaluator</code> is a custom function that implements the <code>KeyedBroadcastProcessFunction</code> interface. It applies the pattern matching logic that we discussed before [...]
-```java
-public static class PatternEvaluator
- extends KeyedBroadcastProcessFunction<Long, Action, Pattern, Tuple2<Long, Pattern>> {
+<div class="highlight"><pre><code class="language-java"><span class="kd">public</span> <span class="kd">static</span> <span class="kd">class</span> <span class="nc">PatternEvaluator</span>
+ <span class="kd">extends</span> <span class="n">KeyedBroadcastProcessFunction</span><span class="o">&lt;</span><span class="n">Long</span><span class="o">,</span> <span class="n">Action</span><span class="o">,</span> <span class="n">Pattern</span><span class="o">,</span> <span class [...]
- // handle for keyed state (per user)
- ValueState<String> prevActionState;
- // broadcast state descriptor
- MapStateDescriptor<Void, Pattern> patternDesc;
+ <span class="c1">// handle for keyed state (per user)</span>
+ <span class="n">ValueState</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">prevActionState</span><span class="o">;</span>
+ <span class="c1">// broadcast state descriptor</span>
+ <span class="n">MapStateDescriptor</span><span class="o">&lt;</span><span class="n">Void</span><span class="o">,</span> <span class="n">Pattern</span><span class="o">&gt;</span> <span class="n">patternDesc</span><span class="o">;</span>
- @Override
- public void open(Configuration conf) {
- // initialize keyed state
- prevActionState = getRuntimeContext().getState(
- new ValueStateDescriptor<>("lastAction", Types.STRING));
- patternDesc =
- new MapStateDescriptor<>("patterns", Types.VOID, Types.POJO(Pattern.class));
- }
-
- /**
- * Called for each user action.
- * Evaluates the current pattern against the previous and
- * current action of the user.
- */
- @Override
- public void processElement(
- Action action,
- ReadOnlyContext ctx,
- Collector<Tuple2<Long, Pattern>> out) throws Exception {
- // get current pattern from broadcast state
- Pattern pattern = ctx
- .getBroadcastState(this.patternDesc)
- // access MapState with null as VOID default value
- .get(null);
- // get previous action of current user from keyed state
- String prevAction = prevActionState.value();
- if (pattern != null && prevAction != null) {
- // user had an action before, check if pattern matches
- if (pattern.firstAction.equals(prevAction) &&
- pattern.secondAction.equals(action.action)) {
- // MATCH
- out.collect(new Tuple2<>(ctx.getCurrentKey(), pattern));
- }
- }
- // update keyed state and remember action for next pattern evaluation
- prevActionState.update(action.action);
- }
-
- /**
- * Called for each new pattern.
- * Overwrites the current pattern with the new pattern.
- */
- @Override
- public void processBroadcastElement(
- Pattern pattern,
- Context ctx,
- Collector<Tuple2<Long, Pattern>> out) throws Exception {
- // store the new pattern by updating the broadcast state
- BroadcastState<Void, Pattern> bcState = ctx.getBroadcastState(patternDesc);
- // storing in MapState with null as VOID default value
- bcState.put(null, pattern);
- }
-}
-```
-
-The `KeyedBroadcastProcessFunction` interface provides three methods to process records and emit results.
-
-- `processBroadcastElement()` is called for each record of the broadcasted stream. In our `PatternEvaluator` function, we simply put the received `Pattern` record in to the broadcast state using the `null` key (remember, we only store a single pattern in the `MapState`).
-- `processElement()` is called for each record of the keyed stream. It provides read-only access to the broadcast state to prevent modification that result in different broadcast states across the parallel instances of the function. The `processElement()` method of the `PatternEvaluator` retrieves the current pattern from the broadcast state and the previous action of the user from the keyed state. If both are present, it checks whether the previous and current action match with the patt [...]
-- `onTimer()` is called when a previously registered timer fires. Timers can be registered in the `processElement` method and are used to perform computations or to clean up state in the future. We did not implement this method in our example to keep the code concise. However, it could be used to remove the last action of a user when the user was not active for a certain period of time to avoid growing state due to inactive users.
-
-You might have noticed the context objects of the `KeyedBroadcastProcessFunction`’s processing method. The context objects give access to additional functionality such as:
-
-- The broadcast state (read-write or read-only, depending on the method),
-- A `TimerService`, which gives access to the record’s timestamp, the current watermark, and which can register timers,
-- The current key (only available in `processElement()`), and
-- A method to apply a function the keyed state of each registered key (only available in `processBroadcastElement()`)
-
-The `KeyedBroadcastProcessFunction` has full access to Flink state and time features just like any other ProcessFunction and hence can be used to implement sophisticated application logic. Broadcast state was designed to be a versatile feature that adapts to different scenarios and use cases. Although we only discussed a fairly simple and restricted application, you can use broadcast state in many ways to implement the requirements of your application.
-
-## Conclusion
-
-In this blog post, we walked you through an example application to explain what Apache Flink’s broadcast state is and how it can be used to evaluate dynamic patterns on event streams. We’ve also discussed the API and showed the source code of our example application.
-
-We invite you to check the [documentation]({{ site.DOCS_BASE_URL }}flink-docs-stable/dev/stream/state/broadcast_state.html) of this feature and provide feedback or suggestions for further improvements through our [mailing list](http://mail-archives.apache.org/mod_mbox/flink-community/).
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="nf">open</span><span class="o">(</span><span class="n">Configuration</span> <span class="n">conf</span><span class="o">)</span> <span class="o">{</span>
+ <span class="c1">// initialize keyed state</span>
+ <span class="n">prevActionState</span> <span class="o">=</span> <span class="n">getRuntimeContext</span><span class="o">().</span><span class="na">getState</span><span class="o">(</span>
+ <span class="k">new</span> <span class="n">ValueStateDescriptor</span><span class="o">&lt;&gt;(</span><span class="s">&quot;lastAction&quot;</span><span class="o">,</span> <span class="n">Types</span><span class="o">.</span><span class="na">STRING</span><span class="o">));</ [...]
+ <span class="n">patternDesc</span> <span class="o">=</span>
+ <span class="k">new</span> <span class="n">MapStateDescriptor</span><span class="o">&lt;&gt;(</span><span class="s">&quot;patterns&quot;</span><span class="o">,</span> <span class="n">Types</span><span class="o">.</span><span class="na">VOID</span><span class="o">,</span> [...]
+ <span class="o">}</span>
+
+ <span class="cm">/**</span>
+<span class="cm"> * Called for each user action.</span>
+<span class="cm"> * Evaluates the current pattern against the previous and</span>
+<span class="cm"> * current action of the user.</span>
+<span class="cm"> */</span>
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="nf">processElement</span><span class="o">(</span>
+ <span class="n">Action</span> <span class="n">action</span><span class="o">,</span>
+ <span class="n">ReadOnlyContext</span> <span class="n">ctx</span><span class="o">,</span>
+ <span class="n">Collector</span><span class="o">&lt;</span><span class="n">Tuple2</span><span class="o">&lt;</span><span class="n">Long</span><span class="o">,</span> <span class="n">Pattern</span><span class="o">&gt;&gt;</span> <span class="n">out</span><span class=&qu [...]
+ <span class="c1">// get current pattern from broadcast state</span>
+ <span class="n">Pattern</span> <span class="n">pattern</span> <span class="o">=</span> <span class="n">ctx</span>
+ <span class="o">.</span><span class="na">getBroadcastState</span><span class="o">(</span><span class="k">this</span><span class="o">.</span><span class="na">patternDesc</span><span class="o">)</span>
+ <span class="c1">// access MapState with null as VOID default value</span>
+ <span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="kc">null</span><span class="o">);</span>
+ <span class="c1">// get previous action of current user from keyed state</span>
+ <span class="n">String</span> <span class="n">prevAction</span> <span class="o">=</span> <span class="n">prevActionState</span><span class="o">.</span><span class="na">value</span><span class="o">();</span>
+ <span class="k">if</span> <span class="o">(</span><span class="n">pattern</span> <span class="o">!=</span> <span class="kc">null</span> <span class="o">&amp;&amp;</span> <span class="n">prevAction</span> <span class="o">!=</span> <span class="kc">null</span><span class="o&qu [...]
+ <span class="c1">// user had an action before, check if pattern matches</span>
+ <span class="k">if</span> <span class="o">(</span><span class="n">pattern</span><span class="o">.</span><span class="na">firstAction</span><span class="o">.</span><span class="na">equals</span><span class="o">(</span><span class="n">prevAction</span><span class="o">)</s [...]
+ <span class="n">pattern</span><span class="o">.</span><span class="na">secondAction</span><span class="o">.</span><span class="na">equals</span><span class="o">(</span><span class="n">action</span><span class="o">.</span><span class="na">action</span><span class="o">)) [...]
+ <span class="c1">// MATCH</span>
+ <span class="n">out</span><span class="o">.</span><span class="na">collect</span><span class="o">(</span><span class="k">new</span> <span class="n">Tuple2</span><span class="o">&lt;&gt;(</span><span class="n">ctx</span><span class="o">.</span><span class="na">get [...]
+ <span class="o">}</span>
+ <span class="o">}</span>
+ <span class="c1">// update keyed state and remember action for next pattern evaluation</span>
+ <span class="n">prevActionState</span><span class="o">.</span><span class="na">update</span><span class="o">(</span><span class="n">action</span><span class="o">.</span><span class="na">action</span><span class="o">);</span>
+ <span class="o">}</span>
+
+ <span class="cm">/**</span>
+<span class="cm"> * Called for each new pattern.</span>
+<span class="cm"> * Overwrites the current pattern with the new pattern.</span>
+<span class="cm"> */</span>
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="nf">processBroadcastElement</span><span class="o">(</span>
+ <span class="n">Pattern</span> <span class="n">pattern</span><span class="o">,</span>
+ <span class="n">Context</span> <span class="n">ctx</span><span class="o">,</span>
+ <span class="n">Collector</span><span class="o">&lt;</span><span class="n">Tuple2</span><span class="o">&lt;</span><span class="n">Long</span><span class="o">,</span> <span class="n">Pattern</span><span class="o">&gt;&gt;</span> <span class="n">out</span><span class=&qu [...]
+ <span class="c1">// store the new pattern by updating the broadcast state</span>
+ <span class="n">BroadcastState</span><span class="o">&lt;</span><span class="n">Void</span><span class="o">,</span> <span class="n">Pattern</span><span class="o">&gt;</span> <span class="n">bcState</span> <span class="o">=</span> <span class="n">ctx</span><span class="o" [...]
+ <span class="c1">// storing in MapState with null as VOID default value</span>
+ <span class="n">bcState</span><span class="o">.</span><span class="na">put</span><span class="o">(</span><span class="kc">null</span><span class="o">,</span> <span class="n">pattern</span><span class="o">);</span>
+ <span class="o">}</span>
+<span class="o">}</span></code></pre></div>
+
+<p>The <code>KeyedBroadcastProcessFunction</code> interface provides three methods to process records and emit results.</p>
+
+<ul>
+ <li><code>processBroadcastElement()</code> is called for each record of the broadcasted stream. In our <code>PatternEvaluator</code> function, we simply put the received <code>Pattern</code> record in to the broadcast state using the <code>null</code> key (remember, we only store a single pattern in the <code>MapState</code>).</li>
+ <li><code>processElement()</code> is called for each record of the keyed stream. It provides read-only access to the broadcast state to prevent modification that result in different broadcast states across the parallel instances of the function. The <code>processElement()</code> method of the <code>PatternEvaluator</code> retrieves the current pattern from the broadcast state and the previous action of the user from the keyed state. If both are [...]
+ <li><code>onTimer()</code> is called when a previously registered timer fires. Timers can be registered in the <code>processElement</code> method and are used to perform computations or to clean up state in the future. We did not implement this method in our example to keep the code concise. However, it could be used to remove the last action of a user when the user was not active for a certain period of time to avoid growing state due to inactive users.&l [...]
+</ul>
+
+<p>You might have noticed the context objects of the <code>KeyedBroadcastProcessFunction</code>’s processing method. The context objects give access to additional functionality such as:</p>
+
+<ul>
+ <li>The broadcast state (read-write or read-only, depending on the method),</li>
+ <li>A <code>TimerService</code>, which gives access to the record’s timestamp, the current watermark, and which can register timers,</li>
+ <li>The current key (only available in <code>processElement()</code>), and</li>
+ <li>A method to apply a function the keyed state of each registered key (only available in <code>processBroadcastElement()</code>)</li>
+</ul>
+
+<p>The <code>KeyedBroadcastProcessFunction</code> has full access to Flink state and time features just like any other ProcessFunction and hence can be used to implement sophisticated application logic. Broadcast state was designed to be a versatile feature that adapts to different scenarios and use cases. Although we only discussed a fairly simple and restricted application, you can use broadcast state in many ways to implement the requirements of your application.</p>
+
+<h2 id="conclusion">Conclusion</h2>
+
+<p>In this blog post, we walked you through an example application to explain what Apache Flink’s broadcast state is and how it can be used to evaluate dynamic patterns on event streams. We’ve also discussed the API and showed the source code of our example application.</p>
+
+<p>We invite you to check the <a href="https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/broadcast_state.html">documentation</a> of this feature and provide feedback or suggestions for further improvements through our <a href="http://mail-archives.apache.org/mod_mbox/flink-community/">mailing list</a>.</p>
</description>
-<pubDate>Wed, 26 Jun 2019 12:00:00 +0000</pubDate>
+<pubDate>Wed, 26 Jun 2019 14:00:00 +0200</pubDate>
<link>https://flink.apache.org/2019/06/26/broadcast-state.html</link>
<guid isPermaLink="true">/2019/06/26/broadcast-state.html</guid>
</item>
@@ -3842,51 +3948,81 @@ We invite you to check the [documentation]({{ site.DOCS_BASE_URL }}flink-docs-st
.tg .tg-center{text-align:center;vertical-align:center}
</style>
-Flink’s network stack is one of the core components that make up the `flink-runtime` module and sit at the heart of every Flink job. It connects individual work units (subtasks) from all TaskManagers. This is where your streamed-in data flows through and it is therefore crucial to the performance of your Flink job for both the throughput as well as latency you observe. In contrast to the coordination channels between TaskManagers and JobManagers which are using RPCs via Akka, the network [...]
-
-This blog post is the first in a series of posts about the network stack. In the sections below, we will first have a high-level look at what abstractions are exposed to the stream operators and then go into detail on the physical implementation and various optimisations Flink did. We will briefly present the result of these optimisations and Flink’s trade-off between throughput and latency. Future blog posts in this series will elaborate more on monitoring and metrics, tuning parameters [...]
+<p>Flink’s network stack is one of the core components that make up the <code>flink-runtime</code> module and sit at the heart of every Flink job. It connects individual work units (subtasks) from all TaskManagers. This is where your streamed-in data flows through and it is therefore crucial to the performance of your Flink job for both the throughput as well as latency you observe. In contrast to the coordination channels between TaskManagers and JobManagers which are [...]
+
+<p>This blog post is the first in a series of posts about the network stack. In the sections below, we will first have a high-level look at what abstractions are exposed to the stream operators and then go into detail on the physical implementation and various optimisations Flink did. We will briefly present the result of these optimisations and Flink’s trade-off between throughput and latency. Future blog posts in this series will elaborate more on monitoring and metrics, tuning p [...]
+
+<div class="page-toc">
+<ul id="markdown-toc">
+ <li><a href="#logical-view" id="markdown-toc-logical-view">Logical View</a></li>
+ <li><a href="#physical-transport" id="markdown-toc-physical-transport">Physical Transport</a> <ul>
+ <li><a href="#inflicting-backpressure-1" id="markdown-toc-inflicting-backpressure-1">Inflicting Backpressure (1)</a></li>
+ </ul>
+ </li>
+ <li><a href="#credit-based-flow-control" id="markdown-toc-credit-based-flow-control">Credit-based Flow Control</a> <ul>
+ <li><a href="#inflicting-backpressure-2" id="markdown-toc-inflicting-backpressure-2">Inflicting Backpressure (2)</a></li>
+ <li><a href="#what-do-we-gain-where-is-the-catch" id="markdown-toc-what-do-we-gain-where-is-the-catch">What do we Gain? Where is the Catch?</a></li>
+ </ul>
+ </li>
+ <li><a href="#writing-records-into-network-buffers-and-reading-them-again" id="markdown-toc-writing-records-into-network-buffers-and-reading-them-again">Writing Records into Network Buffers and Reading them again</a> <ul>
+ <li><a href="#flushing-buffers-to-netty" id="markdown-toc-flushing-buffers-to-netty">Flushing Buffers to Netty</a></li>
+ <li><a href="#buffer-builder--buffer-consumer" id="markdown-toc-buffer-builder--buffer-consumer">Buffer Builder &amp; Buffer Consumer</a></li>
+ </ul>
+ </li>
+ <li><a href="#latency-vs-throughput" id="markdown-toc-latency-vs-throughput">Latency vs. Throughput</a></li>
+ <li><a href="#conclusion" id="markdown-toc-conclusion">Conclusion</a></li>
+</ul>
-{% toc %}
+</div>
-## Logical View
+<h2 id="logical-view">Logical View</h2>
-Flink’s network stack provides the following logical view to the subtasks when communicating with each other, for example during a network shuffle as required by a `keyBy()`.
+<p>Flink’s network stack provides the following logical view to the subtasks when communicating with each other, for example during a network shuffle as required by a <code>keyBy()</code>.</p>
<center>
-<img src="{{ site.baseurl }}/img/blog/2019-06-05-network-stack/flink-network-stack1.png" width="400px" alt="Logical View on Flink's Network Stack"/>
+<img src="/img/blog/2019-06-05-network-stack/flink-network-stack1.png" width="400px" alt="Logical View on Flink's Network Stack" />
</center>
-<br>
-
-It abstracts over the different settings of the following three concepts:
+<p><br /></p>
-* Subtask output type (`ResultPartitionType`):
- * **pipelined (bounded or unbounded):**
- Sending data downstream as soon as it is produced, potentially one-by-one, either as a bounded or unbounded stream of records.
- * **blocking:**
- Sending data downstream only when the full result was produced.
+<p>It abstracts over the different settings of the following three concepts:</p>
-* Scheduling type:
- * **all at once (eager):**
- Deploy all subtasks of the job at the same time (for streaming applications).
- * **next stage on first output (lazy):**
- Deploy downstream tasks as soon as any of their producers generated output.
- * **next stage on complete output:**
- Deploy downstream tasks when any or all of their producers have generated their full output set.
-
-* Transport:
- * **high throughput:**
- Instead of sending each record one-by-one, Flink buffers a bunch of records into its network buffers and sends them altogether. This reduces the costs per record and leads to higher throughput.
- * **low latency via buffer timeout:**
- By reducing the timeout of sending an incompletely filled buffer, you may sacrifice throughput for latency.
+<ul>
+ <li>Subtask output type (<code>ResultPartitionType</code>):
+ <ul>
+ <li><strong>pipelined (bounded or unbounded):</strong>
+ Sending data downstream as soon as it is produced, potentially one-by-one, either as a bounded or unbounded stream of records.</li>
+ <li><strong>blocking:</strong>
+ Sending data downstream only when the full result was produced.</li>
+ </ul>
+ </li>
+ <li>Scheduling type:
+ <ul>
+ <li><strong>all at once (eager):</strong>
+ Deploy all subtasks of the job at the same time (for streaming applications).</li>
+ <li><strong>next stage on first output (lazy):</strong>
+ Deploy downstream tasks as soon as any of their producers generated output.</li>
+ <li><strong>next stage on complete output:</strong>
+ Deploy downstream tasks when any or all of their producers have generated their full output set.</li>
+ </ul>
+ </li>
+ <li>Transport:
+ <ul>
+ <li><strong>high throughput:</strong>
+ Instead of sending each record one-by-one, Flink buffers a bunch of records into its network buffers and sends them altogether. This reduces the costs per record and leads to higher throughput.</li>
+ <li><strong>low latency via buffer timeout:</strong>
+ By reducing the timeout of sending an incompletely filled buffer, you may sacrifice throughput for latency.</li>
+ </ul>
+ </li>
+</ul>
-We will have a look at the throughput and low latency optimisations in the sections below which look at the physical layers of the network stack. For this part, let us elaborate a bit more on the output and scheduling types. First of all, it is important to know that the subtask output type and the scheduling type are closely intertwined making only specific combinations of the two valid.
+<p>We will have a look at the throughput and low latency optimisations in the sections below which look at the physical layers of the network stack. For this part, let us elaborate a bit more on the output and scheduling types. First of all, it is important to know that the subtask output type and the scheduling type are closely intertwined making only specific combinations of the two valid.</p>
-Pipelined result partitions are streaming-style outputs which need a live target subtask to send data to. The target can be scheduled before results are produced or at first output. Batch jobs produce bounded result partitions while streaming jobs produce unbounded results.
+<p>Pipelined result partitions are streaming-style outputs which need a live target subtask to send data to. The target can be scheduled before results are produced or at first output. Batch jobs produce bounded result partitions while streaming jobs produce unbounded results.</p>
-Batch jobs may also produce results in a blocking fashion, depending on the operator and connection pattern that is used. In that case, the complete result must be produced first before the receiving task can be scheduled. This allows batch jobs to work more efficiently and with lower resource usage.
+<p>Batch jobs may also produce results in a blocking fashion, depending on the operator and connection pattern that is used. In that case, the complete result must be produced first before the receiving task can be scheduled. This allows batch jobs to work more efficiently and with lower resource usage.</p>
-The following table summarises the valid combinations:
-<br>
+<p>The following table summarises the valid combinations:
+<br /></p>
<center>
<table class="tg">
<tr>
@@ -3919,24 +4055,22 @@ The following table summarises the valid combinations:
</tr>
</table>
</center>
-<br>
-
+<p><br /></p>
-<sup>1</sup> Currently not used by Flink. <br>
-<sup>2</sup> This may become applicable to streaming jobs once the [Batch/Streaming unification]({{ site.baseurl }}/roadmap.html#batch-and-streaming-unification) is done.
+<p><sup>1</sup> Currently not used by Flink. <br />
+<sup>2</sup> This may become applicable to streaming jobs once the <a href="/roadmap.html#batch-and-streaming-unification">Batch/Streaming unification</a> is done.</p>
+<p><br />
+Additionally, for subtasks with more than one input, scheduling start in two ways: after <em>all</em> or after <em>any</em> input producers to have produced a record/their complete dataset. For tuning the output types and scheduling decisions in batch jobs, please have a look at <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/api/java/org/apache/flink/api/common/ExecutionConfig.html#setExecutionMode-org.apache.flink.api.common.Executio [...]
-<br>
-Additionally, for subtasks with more than one input, scheduling start in two ways: after *all* or after *any* input producers to have produced a record/their complete dataset. For tuning the output types and scheduling decisions in batch jobs, please have a look at [ExecutionConfig#setExecutionMode()]({{ site.DOCS_BASE_URL }}flink-docs-release-1.8/api/java/org/apache/flink/api/common/ExecutionConfig.html#setExecutionMode-org.apache.flink.api.common.ExecutionMode-) - and [ExecutionMode]({ [...]
+<p><br /></p>
-<br>
+<h2 id="physical-transport">Physical Transport</h2>
-## Physical Transport
+<p>In order to understand the physical data connections, please recall that, in Flink, different tasks may share the same slot via <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev/stream/operators/#task-chaining-and-resource-groups">slot sharing groups</a>. TaskManagers may also provide more than one slot to allow multiple subtasks of the same task to be scheduled onto the same TaskManager.</p>
-In order to understand the physical data connections, please recall that, in Flink, different tasks may share the same slot via [slot sharing groups]({{ site.DOCS_BASE_URL }}flink-docs-release-1.8/dev/stream/operators/#task-chaining-and-resource-groups). TaskManagers may also provide more than one slot to allow multiple subtasks of the same task to be scheduled onto the same TaskManager.
-
-For the example pictured below, we will assume a parallelism of 4 and a deployment with two task managers offering 2 slots each. TaskManager 1 executes subtasks A.1, A.2, B.1, and B.2 and TaskManager 2 executes subtasks A.3, A.4, B.3, and B.4. In a shuffle-type connection between task A and task B, for example from a `keyBy()`, there are 2x4 logical connections to handle on each TaskManager, some of which are local, some remote:
-<br>
+<p>For the example pictured below, we will assume a parallelism of 4 and a deployment with two task managers offering 2 slots each. TaskManager 1 executes subtasks A.1, A.2, B.1, and B.2 and TaskManager 2 executes subtasks A.3, A.4, B.3, and B.4. In a shuffle-type connection between task A and task B, for example from a <code>keyBy()</code>, there are 2x4 logical connections to handle on each TaskManager, some of which are local, some remote:
+<br /></p>
<center>
<table class="tg">
@@ -3966,69 +4100,70 @@ For the example pictured below, we will assume a parallelism of 4 and a deployme
</table>
</center>
-<br>
+<p><br /></p>
-Each (remote) network connection between different tasks will get its own TCP channel in Flink’s network stack. However, if different subtasks of the same task are scheduled onto the same TaskManager, their network connections towards the same TaskManagers will be multiplexed and share a single TCP channel for reduced resource usage. In our example, this would apply to A.1 → B.3, A.1 → B.4, as well as A.2 → B.3, and A.2 → B.4 as pictured below:
-<br>
+<p>Each (remote) network connection between different tasks will get its own TCP channel in Flink’s network stack. However, if different subtasks of the same task are scheduled onto the same TaskManager, their network connections towards the same TaskManagers will be multiplexed and share a single TCP channel for reduced resource usage. In our example, this would apply to A.1 → B.3, A.1 → B.4, as well as A.2 → B.3, and A.2 → B.4 as pictured below:
+<br /></p>
<center>
-<img src="{{ site.baseurl }}/img/blog/2019-06-05-network-stack/flink-network-stack2.png" width="700px" alt="Physical-transport-Flink's Network Stack"/>
+<img src="/img/blog/2019-06-05-network-stack/flink-network-stack2.png" width="700px" alt="Physical-transport-Flink's Network Stack" />
</center>
-<br>
+<p><br /></p>
-The results of each subtask are called [ResultPartition]({{ site.DOCS_BASE_URL }}flink-docs-release-1.8/api/java/org/apache/flink/runtime/io/network/partition/ResultPartition.html), each split into separate [ResultSubpartitions]({{ site.DOCS_BASE_URL }}flink-docs-release-1.8/api/java/org/apache/flink/runtime/io/network/partition/ResultSubpartition.html) — one for each logical channel. At this point in the stack, Flink is not dealing with individual records anymore but instead with a grou [...]
+<p>The results of each subtask are called <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/api/java/org/apache/flink/runtime/io/network/partition/ResultPartition.html">ResultPartition</a>, each split into separate <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/api/java/org/apache/flink/runtime/io/network/partition/ResultSubpartition.html">ResultSubpartitions</a> — one for each logical channel. At [...]
- #channels * buffers-per-channel + floating-buffers-per-gate
+<div class="highlight"><pre><code>#channels * buffers-per-channel + floating-buffers-per-gate
+</code></pre></div>
-The total number of buffers on a single TaskManager usually does not need configuration. See the [Configuring the Network Buffers]({{ site.DOCS_BASE_URL }}flink-docs-release-1.8/ops/config.html#configuring-the-network-buffers) documentation for details on how to do so if needed.
+<p>The total number of buffers on a single TaskManager usually does not need configuration. See the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/ops/config.html#configuring-the-network-buffers">Configuring the Network Buffers</a> documentation for details on how to do so if needed.</p>
-### Inflicting Backpressure (1)
+<h3 id="inflicting-backpressure-1">Inflicting Backpressure (1)</h3>
-Whenever a subtask’s sending buffer pool is exhausted — buffers reside in either a result subpartition's buffer queue or inside the lower, Netty-backed network stack — the producer is blocked, cannot continue, and experiences backpressure. The receiver works in a similar fashion: any incoming Netty buffer in the lower network stack needs to be made available to Flink via a network buffer. If there is no network buffer available in the appropriate subtask's buffer pool, Flink will [...]
+<p>Whenever a subtask’s sending buffer pool is exhausted — buffers reside in either a result subpartition’s buffer queue or inside the lower, Netty-backed network stack — the producer is blocked, cannot continue, and experiences backpressure. The receiver works in a similar fashion: any incoming Netty buffer in the lower network stack needs to be made available to Flink via a network buffer. If there is no network buffer available in the appropriate subtask’s buffer pool, Flink wil [...]
-<br>
+<p><br /></p>
<center>
-<img src="{{ site.baseurl }}/img/blog/2019-06-05-network-stack/flink-network-stack3.png" width="700px" alt="Physical-transport-backpressure-Flink's Network Stack"/>
+<img src="/img/blog/2019-06-05-network-stack/flink-network-stack3.png" width="700px" alt="Physical-transport-backpressure-Flink's Network Stack" />
</center>
-<br>
+<p><br /></p>
-To prevent this situation from even happening, Flink 1.5 introduced its own flow control mechanism.
+<p>To prevent this situation from even happening, Flink 1.5 introduced its own flow control mechanism.</p>
-<br>
+<p><br /></p>
-## Credit-based Flow Control
+<h2 id="credit-based-flow-control">Credit-based Flow Control</h2>
-Credit-based flow control makes sure that whatever is “on the wire” will have capacity at the receiver to handle. It is based on the availability of network buffers as a natural extension of the mechanisms Flink had before. Instead of only having a shared local buffer pool, each remote input channel now has its own set of **exclusive buffers**. Conversely, buffers in the local buffer pool are called **floating buffers** as they will float around and are available to every input channel.
+<p>Credit-based flow control makes sure that whatever is “on the wire” will have capacity at the receiver to handle. It is based on the availability of network buffers as a natural extension of the mechanisms Flink had before. Instead of only having a shared local buffer pool, each remote input channel now has its own set of <strong>exclusive buffers</strong>. Conversely, buffers in the local buffer pool are called <strong>floating buffers</strong> as they w [...]
-Receivers will announce the availability of buffers as **credits** to the sender (1 buffer = 1 credit). Each result subpartition will keep track of its **channel credits**. Buffers are only forwarded to the lower network stack if credit is available and each sent buffer reduces the credit score by one. In addition to the buffers, we also send information about the current **backlog** size which specifies how many buffers are waiting in this subpartition’s queue. The receiver will use thi [...]
-<br>
+<p>Receivers will announce the availability of buffers as <strong>credits</strong> to the sender (1 buffer = 1 credit). Each result subpartition will keep track of its <strong>channel credits</strong>. Buffers are only forwarded to the lower network stack if credit is available and each sent buffer reduces the credit score by one. In addition to the buffers, we also send information about the current <strong>backlog</strong> size which specifies [...]
+<br /></p>
<center>
-<img src="{{ site.baseurl }}/img/blog/2019-06-05-network-stack/flink-network-stack4.png" width="700px" alt="Physical-transport-credit-flow-Flink's Network Stack"/>
+<img src="/img/blog/2019-06-05-network-stack/flink-network-stack4.png" width="700px" alt="Physical-transport-credit-flow-Flink's Network Stack" />
</center>
-<br>
+<p><br /></p>
-Credit-based flow control will use [buffers-per-channel]({{ site.DOCS_BASE_URL }}flink-docs-release-1.8/ops/config.html#taskmanager-network-memory-buffers-per-channel) to specify how many buffers are exclusive (mandatory) and [floating-buffers-per-gate]({{ site.DOCS_BASE_URL }}flink-docs-release-1.8/ops/config.html#taskmanager-network-memory-floating-buffers-per-gate) for the local buffer pool (optional<sup>3</sup>) thus achieving the same buffer limit as without flow control [...]
-<br>
+<p>Credit-based flow control will use <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/ops/config.html#taskmanager-network-memory-buffers-per-channel">buffers-per-channel</a> to specify how many buffers are exclusive (mandatory) and <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/ops/config.html#taskmanager-network-memory-floating-buffers-per-gate">floating-buffers-per-gate</a> for the local buffer [...]
+<br /></p>
-<sup>3</sup>If there are not enough buffers available, each buffer pool will get the same share of the globally available ones (± 1).
+<p><sup>3</sup>If there are not enough buffers available, each buffer pool will get the same share of the globally available ones (± 1).</p>
-### Inflicting Backpressure (2)
+<h3 id="inflicting-backpressure-2">Inflicting Backpressure (2)</h3>
-As opposed to the receiver's backpressure mechanisms without flow control, credits provide a more direct control: If a receiver cannot keep up, its available credits will eventually hit 0 and stop the sender from forwarding buffers to the lower network stack. There is backpressure on this logical channel only and there is no need to block reading from a multiplexed TCP channel. Other receivers are therefore not affected in processing available buffers.
+<p>As opposed to the receiver’s backpressure mechanisms without flow control, credits provide a more direct control: If a receiver cannot keep up, its available credits will eventually hit 0 and stop the sender from forwarding buffers to the lower network stack. There is backpressure on this logical channel only and there is no need to block reading from a multiplexed TCP channel. Other receivers are therefore not affected in processing available buffers.</p>
-### What do we Gain? Where is the Catch?
+<h3 id="what-do-we-gain-where-is-the-catch">What do we Gain? Where is the Catch?</h3>
-<img align="right" src="{{ site.baseurl }}/img/blog/2019-06-05-network-stack/flink-network-stack5.png" width="300" height="200" alt="Physical-transport-credit-flow-checkpoints-Flink's Network Stack"/>
+<p><img align="right" src="/img/blog/2019-06-05-network-stack/flink-network-stack5.png" width="300" height="200" alt="Physical-transport-credit-flow-checkpoints-Flink's Network Stack" /></p>
-Since, with flow control, a channel in a multiplex cannot block another of its logical channels, the overall resource utilisation should increase. In addition, by having full control over how much data is “on the wire”, we are also able to improve [checkpoint alignments]({{ site.DOCS_BASE_URL }}flink-docs-release-1.8/internals/stream_checkpointing.html#checkpointing): without flow control, it would take a while for the channel to fill the network stack’s internal buffers and propagate th [...]
+<p>Since, with flow control, a channel in a multiplex cannot block another of its logical channels, the overall resource utilisation should increase. In addition, by having full control over how much data is “on the wire”, we are also able to improve <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/internals/stream_checkpointing.html#checkpointing">checkpoint alignments</a>: without flow control, it would take a while for the channel to fil [...]
-However, the additional announce messages from the receiver may come at some additional costs, especially in setup using SSL-encrypted channels. Also, a single input channel cannot make use of all buffers in the buffer pool because exclusive buffers are not shared. It can also not start right away with sending as much data as is available so that during ramp-up (if you are producing data faster than announcing credits in return) it may take longer to send data through. While this may aff [...]
+<p>However, the additional announce messages from the receiver may come at some additional costs, especially in setup using SSL-encrypted channels. Also, a single input channel cannot make use of all buffers in the buffer pool because exclusive buffers are not shared. It can also not start right away with sending as much data as is available so that during ramp-up (if you are producing data faster than announcing credits in return) it may take longer to send data through. While thi [...]
-There is one more thing you may notice when using credit-based flow control: since we buffer less data between the sender and receiver, you may experience backpressure earlier. This is, however, desired and you do not really get any advantage by buffering more data. If you want to buffer more but keep flow control, you could consider increasing the number of floating buffers via [floating-buffers-per-gate]({{ site.DOCS_BASE_URL }}flink-docs-release-1.8/ops/config.html#taskmanager-network [...]
+<p>There is one more thing you may notice when using credit-based flow control: since we buffer less data between the sender and receiver, you may experience backpressure earlier. This is, however, desired and you do not really get any advantage by buffering more data. If you want to buffer more but keep flow control, you could consider increasing the number of floating buffers via <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/ops/config.html#taskmana [...]
-<br>
+<p><br /></p>
<center>
<table class="tg">
@@ -4038,12 +4173,12 @@ There is one more thing you may notice when using credit-based flow control: sin
</tr>
<tr>
<td class="tg-top">
- • better resource utilisation with data skew in multiplexed connections <br><br>
- • improved checkpoint alignment<br><br>
+ • better resource utilisation with data skew in multiplexed connections <br /><br />
+ • improved checkpoint alignment<br /><br />
• reduced memory use (less data in lower network layers)</td>
<td class="tg-top">
- • additional credit-announce messages<br><br>
- • additional backlog-announce messages (piggy-backed with buffer messages, almost no overhead)<br><br>
+ • additional credit-announce messages<br /><br />
+ • additional backlog-announce messages (piggy-backed with buffer messages, almost no overhead)<br /><br />
• potential round-trip latency</td>
</tr>
<tr>
@@ -4051,967 +4186,1032 @@ There is one more thing you may notice when using credit-based flow control: sin
</tr>
</table>
</center>
-<br>
+<p><br /></p>
-<div class="alert alert-info" markdown="1">
-<span class="label label-info" style="display: inline-block"><span class="glyphicon glyphicon-info-sign" aria-hidden="true"></span> Note</span>
-If you need to turn off credit-based flow control, you can add this to your `flink-conf.yaml`:
+<div class="alert alert-info">
+ <p><span class="label label-info" style="display: inline-block"><span class="glyphicon glyphicon-info-sign" aria-hidden="true"></span> Note</span>
+If you need to turn off credit-based flow control, you can add this to your <code>flink-conf.yaml</code>:</p>
-`taskmanager.network.credit-model: false`
+ <p><code>taskmanager.network.credit-model: false</code></p>
-This parameter, however, is deprecated and will eventually be removed along with the non-credit-based flow control code.
+ <p>This parameter, however, is deprecated and will eventually be removed along with the non-credit-based flow control code.</p>
</div>
-<br>
+<p><br /></p>
-## Writing Records into Network Buffers and Reading them again
+<h2 id="writing-records-into-network-buffers-and-reading-them-again">Writing Records into Network Buffers and Reading them again</h2>
-The following picture extends the slightly more high-level view from above with further details of the network stack and its surrounding components, from the collection of a record in your sending operator to the receiving operator getting it:
-<br>
+<p>The following picture extends the slightly more high-level view from above with further details of the network stack and its surrounding components, from the collection of a record in your sending operator to the receiving operator getting it:
+<br /></p>
<center>
-<img src="{{ site.baseurl }}/img/blog/2019-06-05-network-stack/flink-network-stack6.png" width="700px" alt="Physical-transport-complete-Flink's Network Stack"/>
+<img src="/img/blog/2019-06-05-network-stack/flink-network-stack6.png" width="700px" alt="Physical-transport-complete-Flink's Network Stack" />
</center>
-<br>
+<p><br /></p>
-After creating a record and passing it along, for example via `Collector#collect()`, it is given to the [RecordWriter]({{ site.DOCS_BASE_URL }}flink-docs-release-1.8/api/java/org/apache/flink/runtime/io/network/api/writer/RecordWriter.html) which serialises the record from a Java object into a sequence of bytes which eventually ends up in a network buffer that is handed along as described above. The RecordWriter first serialises the record to a flexible on-heap byte array using the [Span [...]
+<p>After creating a record and passing it along, for example via <code>Collector#collect()</code>, it is given to the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/api/java/org/apache/flink/runtime/io/network/api/writer/RecordWriter.html">RecordWriter</a> which serialises the record from a Java object into a sequence of bytes which eventually ends up in a network buffer that is handed along as described above. The RecordWrite [...]
-On the receiver’s side, the lower network stack (netty) is writing received buffers into the appropriate input channels. The (stream) tasks’s thread eventually reads from these queues and tries to deserialise the accumulated bytes into Java objects with the help of the [RecordReader]({{ site.DOCS_BASE_URL }}flink-docs-release-1.8/api/java/org/apache/flink/runtime/io/network/api/reader/RecordReader.html) and going through the [SpillingAdaptiveSpanningRecordDeserializer]({{ site.DOCS_BASE_ [...]
-<br>
+<p>On the receiver’s side, the lower network stack (netty) is writing received buffers into the appropriate input channels. The (stream) tasks’s thread eventually reads from these queues and tries to deserialise the accumulated bytes into Java objects with the help of the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/api/java/org/apache/flink/runtime/io/network/api/reader/RecordReader.html">RecordReader</a> and going through the <a hr [...]
+<br /></p>
-### Flushing Buffers to Netty
+<h3 id="flushing-buffers-to-netty">Flushing Buffers to Netty</h3>
-In the picture above, the credit-based flow control mechanics actually sit inside the “Netty Server” (and “Netty Client”) components and the buffer the RecordWriter is writing to is always added to the result subpartition in an empty state and then gradually filled with (serialised) records. But when does Netty actually get the buffer? Obviously, it cannot take bytes whenever they become available since that would not only add substantial costs due to cross-thread communication and synch [...]
+<p>In the picture above, the credit-based flow control mechanics actually sit inside the “Netty Server” (and “Netty Client”) components and the buffer the RecordWriter is writing to is always added to the result subpartition in an empty state and then gradually filled with (serialised) records. But when does Netty actually get the buffer? Obviously, it cannot take bytes whenever they become available since that would not only add substantial costs due to cross-thread communication [...]
-In Flink, there are three situations that make a buffer available for consumption by the Netty server:
+<p>In Flink, there are three situations that make a buffer available for consumption by the Netty server:</p>
-* a buffer becomes full when writing a record to it, or<br>
-* the buffer timeout hits, or<br>
-* a special event such as a checkpoint barrier is sent.<br>
-<br>
+<ul>
+ <li>a buffer becomes full when writing a record to it, or<br /></li>
+ <li>the buffer timeout hits, or<br /></li>
+ <li>a special event such as a checkpoint barrier is sent.<br />
+<br /></li>
+</ul>
-#### Flush after Buffer Full
+<h4 id="flush-after-buffer-full">Flush after Buffer Full</h4>
-The RecordWriter works with a local serialisation buffer for the current record and will gradually write these bytes to one or more network buffers sitting at the appropriate result subpartition queue. Although a RecordWriter can work on multiple subpartitions, each subpartition has only one RecordWriter writing data to it. The Netty server, on the other hand, is reading from multiple result subpartitions and multiplexing the appropriate ones into a single channel as described above. Thi [...]
-<br>
+<p>The RecordWriter works with a local serialisation buffer for the current record and will gradually write these bytes to one or more network buffers sitting at the appropriate result subpartition queue. Although a RecordWriter can work on multiple subpartitions, each subpartition has only one RecordWriter writing data to it. The Netty server, on the other hand, is reading from multiple result subpartitions and multiplexing the appropriate ones into a single channel as described a [...]
+<br /></p>
<center>
-<img src="{{ site.baseurl }}/img/blog/2019-06-05-network-stack/flink-network-stack7.png" width="500px" alt="Record-writer-to-network-Flink's Network Stack"/>
+<img src="/img/blog/2019-06-05-network-stack/flink-network-stack7.png" width="500px" alt="Record-writer-to-network-Flink's Network Stack" />
</center>
-<br>
+<p><br /></p>
-<sup>4</sup>We can assume it already got the notification if there are more finished buffers in the queue.
-<br>
+<p><sup>4</sup>We can assume it already got the notification if there are more finished buffers in the queue.
+<br /></p>
-#### Flush after Buffer Timeout
+<h4 id="flush-after-buffer-timeout">Flush after Buffer Timeout</h4>
-In order to support low-latency use cases, we cannot only rely on buffers being full in order to send data downstream. There may be cases where a certain communication channel does not have too many records flowing through and unnecessarily increase the latency of the few records you actually have. Therefore, a periodic process will flush whatever data is available down the stack: the output flusher. The periodic interval can be configured via [StreamExecutionEnvironment#setBufferTimeout [...]
-<br>
+<p>In order to support low-latency use cases, we cannot only rely on buffers being full in order to send data downstream. There may be cases where a certain communication channel does not have too many records flowing through and unnecessarily increase the latency of the few records you actually have. Therefore, a periodic process will flush whatever data is available down the stack: the output flusher. The periodic interval can be configured via <a href="https://ci.apache. [...]
+<br /></p>
<center>
-<img src="{{ site.baseurl }}/img/blog/2019-06-05-network-stack/flink-network-stack8.png" width="500px" alt="Record-writer-to-network-with-flusher-Flink's Network Stack"/>
+<img src="/img/blog/2019-06-05-network-stack/flink-network-stack8.png" width="500px" alt="Record-writer-to-network-with-flusher-Flink's Network Stack" />
</center>
-<br>
+<p><br /></p>
-<sup>5</sup>Strictly speaking, the output flusher does not give any guarantees - it only sends a notification to Netty which can pick it up at will / capacity. This also means that the output flusher has no effect if the channel is backpressured.
-<br>
+<p><sup>5</sup>Strictly speaking, the output flusher does not give any guarantees - it only sends a notification to Netty which can pick it up at will / capacity. This also means that the output flusher has no effect if the channel is backpressured.
+<br /></p>
-#### Flush after special event
+<h4 id="flush-after-special-event">Flush after special event</h4>
-Some special events also trigger immediate flushes if being sent through the RecordWriter. The most important ones are checkpoint barriers or end-of-partition events which obviously should go quickly and not wait for the output flusher to kick in.
-<br>
+<p>Some special events also trigger immediate flushes if being sent through the RecordWriter. The most important ones are checkpoint barriers or end-of-partition events which obviously should go quickly and not wait for the output flusher to kick in.
+<br /></p>
-#### Further remarks
+<h4 id="further-remarks">Further remarks</h4>
-In contrast to Flink < 1.5, please note that (a) network buffers are now placed in the subpartition queues directly and (b) we are not closing the buffer on each flush. This gives us a few advantages:
+<p>In contrast to Flink &lt; 1.5, please note that (a) network buffers are now placed in the subpartition queues directly and (b) we are not closing the buffer on each flush. This gives us a few advantages:</p>
-* less synchronisation overhead (output flusher and RecordWriter are independent)
-* in high-load scenarios where Netty is the bottleneck (either through backpressure or directly), we can still accumulate data in incomplete buffers
-* significant reduction of Netty notifications
+<ul>
+ <li>less synchronisation overhead (output flusher and RecordWriter are independent)</li>
+ <li>in high-load scenarios where Netty is the bottleneck (either through backpressure or directly), we can still accumulate data in incomplete buffers</li>
+ <li>significant reduction of Netty notifications</li>
+</ul>
-However, you may notice an increased CPU use and TCP packet rate during low load scenarios. This is because, with the changes, Flink will use any *available* CPU cycles to try to maintain the desired latency. Once the load increases, this will self-adjust by buffers filling up more. High load scenarios are not affected and even get a better throughput because of the reduced synchronisation overhead.
-<br>
+<p>However, you may notice an increased CPU use and TCP packet rate during low load scenarios. This is because, with the changes, Flink will use any <em>available</em> CPU cycles to try to maintain the desired latency. Once the load increases, this will self-adjust by buffers filling up more. High load scenarios are not affected and even get a better throughput because of the reduced synchronisation overhead.
+<br /></p>
-### Buffer Builder & Buffer Consumer
+<h3 id="buffer-builder--buffer-consumer">Buffer Builder &amp; Buffer Consumer</h3>
-If you want to dig deeper into how the producer-consumer mechanics are implemented in Flink, please take a closer look at the [BufferBuilder]({{ site.DOCS_BASE_URL }}flink-docs-release-1.8/api/java/org/apache/flink/runtime/io/network/buffer/BufferBuilder.html) and [BufferConsumer]({{ site.DOCS_BASE_URL }}flink-docs-release-1.8/api/java/org/apache/flink/runtime/io/network/buffer/BufferConsumer.html) classes which have been introduced in Flink 1.5. While reading is potentially only *per bu [...]
+<p>If you want to dig deeper into how the producer-consumer mechanics are implemented in Flink, please take a closer look at the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/api/java/org/apache/flink/runtime/io/network/buffer/BufferBuilder.html">BufferBuilder</a> and <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/api/java/org/apache/flink/runtime/io/network/buffer/BufferConsumer.html">BufferConsumer [...]
-<br>
+<p><br /></p>
-## Latency vs. Throughput
+<h2 id="latency-vs-throughput">Latency vs. Throughput</h2>
-Network buffers were introduced to get higher resource utilisation and higher throughput at the cost of having some records wait in buffers a little longer. Although an upper limit to this wait time can be given via the buffer timeout, you may be curious to find out more about the trade-off between these two dimensions: latency and throughput, as, obviously, you cannot get both. The following plot shows various values for the buffer timeout starting at 0 (flush with every record) to 100m [...]
-<br>
+<p>Network buffers were introduced to get higher resource utilisation and higher throughput at the cost of having some records wait in buffers a little longer. Although an upper limit to this wait time can be given via the buffer timeout, you may be curious to find out more about the trade-off between these two dimensions: latency and throughput, as, obviously, you cannot get both. The following plot shows various values for the buffer timeout starting at 0 (flush with every record [...]
+<br /></p>
<center>
-<img src="{{ site.baseurl }}/img/blog/2019-06-05-network-stack/flink-network-stack9.png" width="650px" alt="Network-buffertimeout-Flink's Network Stack"/>
+<img src="/img/blog/2019-06-05-network-stack/flink-network-stack9.png" width="650px" alt="Network-buffertimeout-Flink's Network Stack" />
</center>
-<br>
-
-As you can see, with Flink 1.5+, even very low buffer timeouts such as 1ms (for low-latency scenarios) provide a maximum throughput as high as 75% of the default timeout where more data is buffered before being sent over the wire.
+<p><br /></p>
-<br>
+<p>As you can see, with Flink 1.5+, even very low buffer timeouts such as 1ms (for low-latency scenarios) provide a maximum throughput as high as 75% of the default timeout where more data is buffered before being sent over the wire.</p>
-## Conclusion
+<p><br /></p>
-Now you know about result partitions, the different network connections and scheduling types for both batch and streaming. You also know about credit-based flow control and how the network stack works internally, in order to reason about network-related tuning parameters and about certain job behaviours. Future blog posts in this series will build upon this knowledge and go into more operational details including relevant metrics to look at, further network stack tuning, and common antip [...]
+<h2 id="conclusion">Conclusion</h2>
+<p>Now you know about result partitions, the different network connections and scheduling types for both batch and streaming. You also know about credit-based flow control and how the network stack works internally, in order to reason about network-related tuning parameters and about certain job behaviours. Future blog posts in this series will build upon this knowledge and go into more operational details including relevant metrics to look at, further network stack tuning, and com [...]
</description>
-<pubDate>Wed, 05 Jun 2019 08:45:00 +0000</pubDate>
+<pubDate>Wed, 05 Jun 2019 10:45:00 +0200</pubDate>
<link>https://flink.apache.org/2019/06/05/flink-network-stack.html</link>
<guid isPermaLink="true">/2019/06/05/flink-network-stack.html</guid>
</item>
<item>
<title>State TTL in Flink 1.8.0: How to Automatically Cleanup Application State in Apache Flink</title>
-<description>A common requirement for many stateful streaming applications is to automatically cleanup application state for effective management of your state size, or to control how long the application state can be accessed (e.g. due to legal regulations like the GDPR). The state time-to-live (TTL) feature was initiated in Flink 1.6.0 and enabled application state cleanup and efficient state size management in Apache Flink.
+<description><p>A common requirement for many stateful streaming applications is to automatically cleanup application state for effective management of your state size, or to control how long the application state can be accessed (e.g. due to legal regulations like the GDPR). The state time-to-live (TTL) feature was initiated in Flink 1.6.0 and enabled application state cleanup and efficient state size management in Apache Flink.</p>
-In this post, we motivate the State TTL feature and discuss its use cases. Moreover, we show how to use and configure it. We explain how Flink internally manages state with TTL and present some exciting additions to the feature in Flink 1.8.0. The blog post concludes with an outlook on future improvements and extensions.
+<p>In this post, we motivate the State TTL feature and discuss its use cases. Moreover, we show how to use and configure it. We explain how Flink internally manages state with TTL and present some exciting additions to the feature in Flink 1.8.0. The blog post concludes with an outlook on future improvements and extensions.</p>
-# The Transient Nature of State
+<h1 id="the-transient-nature-of-state">The Transient Nature of State</h1>
-There are two major reasons why state should be maintained only for a limited time. For example, let’s imagine a Flink application that ingests a stream of user login events and stores for each user the time of the last login to improve the experience of frequent visitors.
+<p>There are two major reasons why state should be maintained only for a limited time. For example, let’s imagine a Flink application that ingests a stream of user login events and stores for each user the time of the last login to improve the experience of frequent visitors.</p>
-* **Controlling the size of state.**
-Being able to efficiently manage an ever-growing state size is a primary use case for state TTL. Oftentimes, data needs to be persisted temporarily while there is some user activity around it, e.g. web sessions. When the activity ends there is no longer interest in that data while it still occupies storage. Flink 1.8.0 introduces background cleanup of old state based on TTL that makes the eviction of no-longer-necessary data frictionless. Previously, the application developer had to take [...]
-
-* **Complying with data protection and sensitive data requirements.**
-Recent developments around data privacy regulations, such as the General Data Protection Regulation (GDPR) introduced by the European Union, make compliance with such data requirements or treating sensitive data a top priority for many use cases and applications. An example of such use cases includes applications that require keeping data for a specific timeframe and preventing access to it thereafter. This is a common challenge for companies providing short-term services to their custom [...]
+<ul>
+ <li>
+ <p><strong>Controlling the size of state.</strong>
+Being able to efficiently manage an ever-growing state size is a primary use case for state TTL. Oftentimes, data needs to be persisted temporarily while there is some user activity around it, e.g. web sessions. When the activity ends there is no longer interest in that data while it still occupies storage. Flink 1.8.0 introduces background cleanup of old state based on TTL that makes the eviction of no-longer-necessary data frictionless. Previously, the application developer had to take [...]
+ </li>
+ <li>
+ <p><strong>Complying with data protection and sensitive data requirements.</strong>
+Recent developments around data privacy regulations, such as the General Data Protection Regulation (GDPR) introduced by the European Union, make compliance with such data requirements or treating sensitive data a top priority for many use cases and applications. An example of such use cases includes applications that require keeping data for a specific timeframe and preventing access to it thereafter. This is a common challenge for companies providing short-term services to their custom [...]
+ </li>
+</ul>
-Both requirements can be addressed by a feature that periodically, yet continuously, removes the state for a key once it becomes unnecessary or unimportant and there is no requirement to keep it in storage any more.
+<p>Both requirements can be addressed by a feature that periodically, yet continuously, removes the state for a key once it becomes unnecessary or unimportant and there is no requirement to keep it in storage any more.</p>
-# State TTL for continuous cleanup of application state
+<h1 id="state-ttl-for-continuous-cleanup-of-application-state">State TTL for continuous cleanup of application state</h1>
-The 1.6.0 release of Apache Flink introduced the State TTL feature. It enabled developers of stream processing applications to configure the state of operators to expire and be cleaned up after a defined timeout (time-to-live). In Flink 1.8.0 the feature was extended, including continuous cleanup of old entries for both the RocksDB and the heap state backends (FSStateBackend and MemoryStateBackend), enabling a continuous cleanup process of old entries (according to the TTL setting).
+<p>The 1.6.0 release of Apache Flink introduced the State TTL feature. It enabled developers of stream processing applications to configure the state of operators to expire and be cleaned up after a defined timeout (time-to-live). In Flink 1.8.0 the feature was extended, including continuous cleanup of old entries for both the RocksDB and the heap state backends (FSStateBackend and MemoryStateBackend), enabling a continuous cleanup process of old entries (according to the TTL setti [...]
-In Flink’s DataStream API, application state is defined by a [state descriptor]({{ site.DOCS_BASE_URL }}flink-docs-release-1.8/dev/stream/state/state.html#using-managed-keyed-state). State TTL is configured by passing a `StateTtlConfiguration` object to a state descriptor. The following Java example shows how to create a state TTL configuration and provide it to the state descriptor that holds the last login time of a user as a `Long` value:
+<p>In Flink’s DataStream API, application state is defined by a <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev/stream/state/state.html#using-managed-keyed-state">state descriptor</a>. State TTL is configured by passing a <code>StateTtlConfiguration</code> object to a state descriptor. The following Java example shows how to create a state TTL configuration and provide it to the state descriptor that holds the last login ti [...]
-```java
-import org.apache.flink.api.common.state.StateTtlConfig;
-import org.apache.flink.api.common.time.Time;
-import org.apache.flink.api.common.state.ValueStateDescriptor;
+<div class="highlight"><pre><code class="language-java"><span class="kn">import</span> <span class="nn">org.apache.flink.api.common.state.StateTtlConfig</span><span class="o">;</span>
+<span class="kn">import</span> <span class="nn">org.apache.flink.api.common.time.Time</span><span class="o">;</span>
+<span class="kn">import</span> <span class="nn">org.apache.flink.api.common.state.ValueStateDescriptor</span><span class="o">;</span>
-StateTtlConfig ttlConfig = StateTtlConfig
- .newBuilder(Time.days(7))
- .setUpdateType(StateTtlConfig.UpdateType.OnCreateAndWrite)
- .setStateVisibility(StateTtlConfig.StateVisibility.NeverReturnExpired)
- .build();
+<span class="n">StateTtlConfig</span> <span class="n">ttlConfig</span> <span class="o">=</span> <span class="n">StateTtlConfig</span>
+ <span class="o">.</span><span class="na">newBuilder</span><span class="o">(</span><span class="n">Time</span><span class="o">.</span><span class="na">days</span><span class="o">(</span><span class="mi">7</span><span class="o">))</span>
+ <span class="o">.</span><span class="na">setUpdateType</span><span class="o">(</span><span class="n">StateTtlConfig</span><span class="o">.</span><span class="na">UpdateType</span><span class="o">.</span><span class="na">OnCreateAndWrite</span><span class="o">)</span>
+ <span class="o">.</span><span class="na">setStateVisibility</span><span class="o">(</span><span class="n">StateTtlConfig</span><span class="o">.</span><span class="na">StateVisibility</span><span class="o">.</span><span class="na">NeverReturnExpired</span><span class="o">)</span>
+ <span class="o">.</span><span class="na">build</span><span class="o">();</span>
-ValueStateDescriptor<Long> lastUserLogin =
- new ValueStateDescriptor<>("lastUserLogin", Long.class);
+<span class="n">ValueStateDescriptor</span><span class="o">&lt;</span><span class="n">Long</span><span class="o">&gt;</span> <span class="n">lastUserLogin</span> <span class="o">=</span>
+ <span class="k">new</span> <span class="n">ValueStateDescriptor</span><span class="o">&lt;&gt;(</span><span class="s">&quot;lastUserLogin&quot;</span><span class="o">,</span> <span class="n">Long</span><span class="o">.</span><span class="na">class</span><span class="o">);</span>
-lastUserLogin.enableTimeToLive(ttlConfig);
-```
+<span class="n">lastUserLogin</span><span class="o">.</span><span class="na">enableTimeToLive</span><span class="o">(</span><span class="n">ttlConfig</span><span class="o">);</span></code></pre></div>
-Flink provides multiple options to configure the behavior of the state TTL functionality.
+<p>Flink provides multiple options to configure the behavior of the state TTL functionality.</p>
-* **When is the Time-to-Live reset?**
-By default, the expiration time of a state entry is updated when the state is modified. Optionally, it can also be updated on read access at the cost of an additional write operation to update the timestamp.
-
-* **Can the expired state be accessed one last time?**
-State TTL employs a lazy strategy to clean up expired state. This can lead to the situation that an application attempts to read state which is expired but hasn’t been removed yet. You can configure whether such a read request returns the expired state or not. In either case, the expired state is immediately removed afterwards. While the option of returning expired state favors data availability, not returning expired state can be required for data protection regulations.
-
-* **Which time semantics are used for the Time-to-Live timers?**
-With Flink 1.8.0, users can only define a state TTL in terms of processing time. The support for event time is planned for future Apache Flink releases.
+<ul>
+ <li>
+ <p><strong>When is the Time-to-Live reset?</strong>
+By default, the expiration time of a state entry is updated when the state is modified. Optionally, it can also be updated on read access at the cost of an additional write operation to update the timestamp.</p>
+ </li>
+ <li>
+ <p><strong>Can the expired state be accessed one last time?</strong>
+State TTL employs a lazy strategy to clean up expired state. This can lead to the situation that an application attempts to read state which is expired but hasn’t been removed yet. You can configure whether such a read request returns the expired state or not. In either case, the expired state is immediately removed afterwards. While the option of returning expired state favors data availability, not returning expired state can be required for data protection regulations.</p>
+ </li>
+ <li>
+ <p><strong>Which time semantics are used for the Time-to-Live timers?</strong>
+With Flink 1.8.0, users can only define a state TTL in terms of processing time. The support for event time is planned for future Apache Flink releases.</p>
+ </li>
+</ul>
-You can read more about how to use state TTL in the [Apache Flink documentation]({{ site.DOCS_BASE_URL }}flink-docs-stable/dev/stream/state/state.html#state-time-to-live-ttl).
+<p>You can read more about how to use state TTL in the <a href="https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/state.html#state-time-to-live-ttl">Apache Flink documentation</a>.</p>
-Internally, the State TTL feature is implemented by storing an additional timestamp of the last relevant state access, along with the actual state value. While this approach adds some storage overhead, it allows Flink to check for the expired state during state access, checkpointing, recovery, or dedicated storage cleanup procedures.
+<p>Internally, the State TTL feature is implemented by storing an additional timestamp of the last relevant state access, along with the actual state value. While this approach adds some storage overhead, it allows Flink to check for the expired state during state access, checkpointing, recovery, or dedicated storage cleanup procedures.</p>
-# “Taking out the Garbage”
+<h1 id="taking-out-the-garbage">“Taking out the Garbage”</h1>
-When a state object is accessed in a read operation, Flink will check its timestamp and clear the state if it is expired (depending on the configured state visibility, the expired state is returned or not). Due to this lazy removal, expired state that is never accessed again will forever occupy storage space unless it is garbage collected.
+<p>When a state object is accessed in a read operation, Flink will check its timestamp and clear the state if it is expired (depending on the configured state visibility, the expired state is returned or not). Due to this lazy removal, expired state that is never accessed again will forever occupy storage space unless it is garbage collected.</p>
-So how can the expired state be removed without the application logic explicitly taking care of it? In general, there are different possible strategies to remove it in the background.
+<p>So how can the expired state be removed without the application logic explicitly taking care of it? In general, there are different possible strategies to remove it in the background.</p>
-## Keep full state snapshots clean
+<h2 id="keep-full-state-snapshots-clean">Keep full state snapshots clean</h2>
-Flink 1.6.0 already supported automatic eviction of the expired state when a full snapshot for a checkpoint or savepoint is taken. Note that state eviction is not applied for incremental checkpoints. State eviction on full snapshots must be explicitly enabled as shown in the following example:
+<p>Flink 1.6.0 already supported automatic eviction of the expired state when a full snapshot for a checkpoint or savepoint is taken. Note that state eviction is not applied for incremental checkpoints. State eviction on full snapshots must be explicitly enabled as shown in the following example:</p>
-```java
-StateTtlConfig ttlConfig = StateTtlConfig
- .newBuilder(Time.days(7))
- .cleanupFullSnapshot()
- .build();
-```
+<div class="highlight"><pre><code class="language-java"><span class="n">StateTtlConfig</span> <span class="n">ttlConfig</span> <span class="o">=</span> <span class="n">StateTtlConfig</span>
+ <span class="o">.</span><span class="na">newBuilder</span><span class="o">(</span><span class="n">Time</span><span class="o">.</span><span class="na">days</span><span class="o">(</span><span class="mi">7</span><span class="o">))</span>
+ <span class="o">.</span><span class="na">cleanupFullSnapshot</span><span class="o">()</span>
+ <span class="o">.</span><span class="na">build</span><span class="o">();</span></code></pre></div>
-The local storage stays untouched but the size of the stored snapshot is reduced. The local state of an operator will only be cleaned up when the operator reloads its state from a snapshot, i.e. in case of recovery or when starting from a savepoint.
+<p>The local storage stays untouched but the size of the stored snapshot is reduced. The local state of an operator will only be cleaned up when the operator reloads its state from a snapshot, i.e. in case of recovery or when starting from a savepoint.</p>
-Due to these limitations, applications still need to actively remove state after it expired in Flink 1.6.0. To improve the user experience, Flink 1.8.0 introduces two more autonomous cleanup strategies, one for each of Flink’s two state backend types. We describe them below.
+<p>Due to these limitations, applications still need to actively remove state after it expired in Flink 1.6.0. To improve the user experience, Flink 1.8.0 introduces two more autonomous cleanup strategies, one for each of Flink’s two state backend types. We describe them below.</p>
-## Incremental cleanup in Heap state backends
+<h2 id="incremental-cleanup-in-heap-state-backends">Incremental cleanup in Heap state backends</h2>
-This approach is specific to the Heap state backends (FSStateBackend and MemoryStateBackend). The idea is that the storage backend keeps a lazy global iterator over all state entries. Certain events, for instance state access, trigger an incremental cleanup. Every time an incremental cleanup is triggered, the iterator is advanced. The traversed state entries are checked and expired once are removed. The following code example shows how to enable incremental cleanup:
+<p>This approach is specific to the Heap state backends (FSStateBackend and MemoryStateBackend). The idea is that the storage backend keeps a lazy global iterator over all state entries. Certain events, for instance state access, trigger an incremental cleanup. Every time an incremental cleanup is triggered, the iterator is advanced. The traversed state entries are checked and expired once are removed. The following code example shows how to enable incremental cleanup:</p>
-```java
-StateTtlConfig ttlConfig = StateTtlConfig
- .newBuilder(Time.days(7))
- // check 10 keys for every state access
- .cleanupIncrementally(10, false)
- .build();
-```
+<div class="highlight"><pre><code class="language-java"><span class="n">StateTtlConfig</span> <span class="n">ttlConfig</span> <span class="o">=</span> <span class="n">StateTtlConfig</span>
+ <span class="o">.</span><span class="na">newBuilder</span><span class="o">(</span><span class="n">Time</span><span class="o">.</span><span class="na">days</span><span class="o">(</span><span class="mi">7</span><span class="o">))</span>
+ <span class="c1">// check 10 keys for every state access</span>
+ <span class="o">.</span><span class="na">cleanupIncrementally</span><span class="o">(</span><span class="mi">10</span><span class="o">,</span> <span class="kc">false</span><span class="o">)</span>
+ <span class="o">.</span><span class="na">build</span><span class="o">();</span></code></pre></div>
-If enabled, every state access triggers a cleanup step. For every clean up step, a certain number of state entries are checked for expiration. There are two tuning parameters. The first defines the number of state entries to check for each cleanup step. The second parameter is a flag to trigger a cleanup step after each processed record, additionally to each state access.
+<p>If enabled, every state access triggers a cleanup step. For every clean up step, a certain number of state entries are checked for expiration. There are two tuning parameters. The first defines the number of state entries to check for each cleanup step. The second parameter is a flag to trigger a cleanup step after each processed record, additionally to each state access.</p>
-There are two important caveats about this approach:
+<p>There are two important caveats about this approach:
* The first one is that the time spent for the incremental cleanup increases the record processing latency.
-* The second one should be practically negligible but still worth mentioning: if no state is accessed or no records are processed, expired state won’t be removed.
+* The second one should be practically negligible but still worth mentioning: if no state is accessed or no records are processed, expired state won’t be removed.</p>
-## RocksDB background compaction to filter out expired state
+<h2 id="rocksdb-background-compaction-to-filter-out-expired-state">RocksDB background compaction to filter out expired state</h2>
-If your application uses the RocksDB state backend, you can enable another cleanup strategy which is based on a Flink specific compaction filter. RocksDB periodically runs asynchronous compactions to merge state updates and reduce storage. The Flink compaction filter checks the expiration timestamp of state entries with TTL and discards all expired values.
+<p>If your application uses the RocksDB state backend, you can enable another cleanup strategy which is based on a Flink specific compaction filter. RocksDB periodically runs asynchronous compactions to merge state updates and reduce storage. The Flink compaction filter checks the expiration timestamp of state entries with TTL and discards all expired values.</p>
-The first step to activate this feature is to configure the RocksDB state backend by setting the following Flink configuration option: `state.backend.rocksdb.ttl.compaction.filter.enabled`. Once the RocksDB state backend is configured, the compaction cleanup strategy is enabled for a state as shown in the following code example:
+<p>The first step to activate this feature is to configure the RocksDB state backend by setting the following Flink configuration option: <code>state.backend.rocksdb.ttl.compaction.filter.enabled</code>. Once the RocksDB state backend is configured, the compaction cleanup strategy is enabled for a state as shown in the following code example:</p>
-```java
-StateTtlConfig ttlConfig = StateTtlConfig
- .newBuilder(Time.days(7))
- .cleanupInRocksdbCompactFilter()
- .build();
-```
-Keep in mind that calling the Flink TTL filter slows down the RocksDB compaction.
+<div class="highlight"><pre><code class="language-java"><span class="n">StateTtlConfig</span> <span class="n">ttlConfig</span> <span class="o">=</span> <span class="n">StateTtlConfig</span>
+ <span class="o">.</span><span class="na">newBuilder</span><span class="o">(</span><span class="n">Time</span><span class="o">.</span><span class="na">days</span><span class="o">(</span><span class="mi">7</span><span class="o">))</span>
+ <span class="o">.</span><span class="na">cleanupInRocksdbCompactFilter</span><span class="o">()</span>
+ <span class="o">.</span><span class="na">build</span><span class="o">();</span></code></pre></div>
+<p>Keep in mind that calling the Flink TTL filter slows down the RocksDB compaction.</p>
-## Eager State Cleanup with Timers
+<h2 id="eager-state-cleanup-with-timers">Eager State Cleanup with Timers</h2>
-Another way to manually cleanup state is based on Flink timers. This is an idea that the community is currently evaluating for future releases. With this approach, a cleanup timer is registered for every state access. This approach is more predictable because state is eagerly removed as soon as it expires. However, it is more expensive because the timers consume storage along with the original state.
+<p>Another way to manually cleanup state is based on Flink timers. This is an idea that the community is currently evaluating for future releases. With this approach, a cleanup timer is registered for every state access. This approach is more predictable because state is eagerly removed as soon as it expires. However, it is more expensive because the timers consume storage along with the original state.</p>
-# Future work
+<h1 id="future-work">Future work</h1>
-Apart from including the timer-based cleanup strategy, mentioned above, the Flink community has plans to further improve the state TTL feature. The possible improvements include adding support of TTL for event time scale (only processing time is supported at the moment) and enabling State TTL for queryable state.
+<p>Apart from including the timer-based cleanup strategy, mentioned above, the Flink community has plans to further improve the state TTL feature. The possible improvements include adding support of TTL for event time scale (only processing time is supported at the moment) and enabling State TTL for queryable state.</p>
-We encourage you to join the conversation and share your thoughts and ideas in the [Apache Flink JIRA board](https://issues.apache.org/jira/projects/FLINK/summary) or by subscribing to the Apache Flink dev mailing list. Feedback or suggestions are always appreciated and we look forward to hearing your thoughts on the Flink mailing lists.
+<p>We encourage you to join the conversation and share your thoughts and ideas in the <a href="https://issues.apache.org/jira/projects/FLINK/summary">Apache Flink JIRA board</a> or by subscribing to the Apache Flink dev mailing list. Feedback or suggestions are always appreciated and we look forward to hearing your thoughts on the Flink mailing lists.</p>
-# Summary
+<h1 id="summary">Summary</h1>
-Time-based state access restrictions and controlling the size of application state are common challenges in the world of stateful stream processing. Flink’s 1.8.0 release significantly improves the State TTL feature by adding support for continuous background cleanup of expired state objects. The new clean up mechanisms relieve you from manually implementing state cleanup. They are also more efficient due to their lazy nature. State TTL gives you control over the size of your application [...]
+<p>Time-based state access restrictions and controlling the size of application state are common challenges in the world of stateful stream processing. Flink’s 1.8.0 release significantly improves the State TTL feature by adding support for continuous background cleanup of expired state objects. The new clean up mechanisms relieve you from manually implementing state cleanup. They are also more efficient due to their lazy nature. State TTL gives you control over the size of your ap [...]
</description>
-<pubDate>Sun, 19 May 2019 12:00:00 +0000</pubDate>
+<pubDate>Sun, 19 May 2019 14:00:00 +0200</pubDate>
<link>https://flink.apache.org/2019/05/19/state-ttl.html</link>
<guid isPermaLink="true">/2019/05/19/state-ttl.html</guid>
</item>
<item>
<title>Flux capacitor, huh? Temporal Tables and Joins in Streaming SQL</title>
-<description>Figuring out how to manage and model temporal data for effective point-in-time analysis was a longstanding battle, dating as far back as the early 80’s, that culminated with the introduction of temporal tables in the SQL standard in 2011. Up to that point, users were doomed to implement this as part of the application logic, often hurting the length of the development lifecycle as well as the maintainability of the code. And, although there isn’t a single, commonly accepted [...]
+<description><p>Figuring out how to manage and model temporal data for effective point-in-time analysis was a longstanding battle, dating as far back as the early 80’s, that culminated with the introduction of temporal tables in the SQL standard in 2011. Up to that point, users were doomed to implement this as part of the application logic, often hurting the length of the development lifecycle as well as the maintainability of the code. And, although there isn’t a single, commonly [...]
<center>
-<img src="{{ site.baseurl }}/img/blog/2019-05-13-temporal-tables/TemporalTables1.png" width="500px" alt="Taxi Fares and Conversion Rates"/>
+<img src="/img/blog/2019-05-13-temporal-tables/TemporalTables1.png" width="500px" alt="Taxi Fares and Conversion Rates" />
</center>
-<br>
+<p><br /></p>
-**For example:** given a stream with Taxi Fare events tied to the local currency of the ride location, we might want to convert the fare price to a common currency for further processing. As conversion rates excel at fluctuating over time, each Taxi Fare event would need to be matched to the rate that was valid at the time the event occurred in order to produce a reliable result.
+<p><strong>For example:</strong> given a stream with Taxi Fare events tied to the local currency of the ride location, we might want to convert the fare price to a common currency for further processing. As conversion rates excel at fluctuating over time, each Taxi Fare event would need to be matched to the rate that was valid at the time the event occurred in order to produce a reliable result.</p>
-## Modelling Temporal Data with Flink
+<h2 id="modelling-temporal-data-with-flink">Modelling Temporal Data with Flink</h2>
-In the 1.7 release, Flink has introduced the concept of **temporal tables** into its streaming SQL and Table API: parameterized views on append-only tables — or, any table that only allows records to be inserted, never updated or deleted — that are interpreted as a changelog and keep data closely tied to time context, so that it can be interpreted as valid only within a specific period of time. Transforming a stream into a temporal table requires:
+<p>In the 1.7 release, Flink has introduced the concept of <strong>temporal tables</strong> into its streaming SQL and Table API: parameterized views on append-only tables — or, any table that only allows records to be inserted, never updated or deleted — that are interpreted as a changelog and keep data closely tied to time context, so that it can be interpreted as valid only within a specific period of time. Transforming a stream into a temporal table requires:</p>
-* Defining a **primary key** and a **versioning field** that can be used to keep track of the changes that happen over time;
-
-* Exposing the stream as a **temporal table function** that maps each point in time to a static relation.
+<ul>
+ <li>
+ <p>Defining a <strong>primary key</strong> and a <strong>versioning field</strong> that can be used to keep track of the changes that happen over time;</p>
+ </li>
+ <li>
+ <p>Exposing the stream as a <strong>temporal table function</strong> that maps each point in time to a static relation.</p>
+ </li>
+</ul>
-Going back to our example use case, a temporal table is just what we need to model the conversion rate data such as to make it useful for point-in-time querying. Temporal table functions are implemented as an extension of Flink’s generic [table function]({{ site.DOCS_BASE_URL }}flink-docs-release-1.8/dev/table/udfs.html#table-functions) class and can be defined in the same straightforward way to be used with the Table API or SQL parser.
+<p>Going back to our example use case, a temporal table is just what we need to model the conversion rate data such as to make it useful for point-in-time querying. Temporal table functions are implemented as an extension of Flink’s generic <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev/table/udfs.html#table-functions">table function</a> class and can be defined in the same straightforward way to be used with the Table API or SQL pars [...]
-```java
-import org.apache.flink.table.functions.TemporalTableFunction;
+<div class="highlight"><pre><code class="language-java"><span class="kn">import</span> <span class="nn">org.apache.flink.table.functions.TemporalTableFunction</span><span class="o">;</span>
-(...)
+<span class="o">(...)</span>
-// Get the stream and table environments.
-StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
-StreamTableEnvironment tEnv = StreamTableEnvironment.getTableEnvironment(env);
+<span class="c1">// Get the stream and table environments.</span>
+<span class="n">StreamExecutionEnvironment</span> <span class="n">env</span> <span class="o">=</span> <span class="n">StreamExecutionEnvironment</span><span class="o">.</span><span class="na">getExecutionEnvironment</span><span class="o">();</span>
+<span class="n">StreamTableEnvironment</span> <span class="n">tEnv</span> <span class="o">=</span> <span class="n">StreamTableEnvironment</span><span class="o">.</span><span class="na">getTableEnvironment</span><span class="o">(</span><span class="n">env</span><span class="o">);</span>
-// Provide a sample static data set of the rates history table.
-List <Tuple2<String, Long>>ratesHistoryData =new ArrayList<>();
+<span class="c1">// Provide a sample static data set of the rates history table.</span>
+<span class="n">List</span> <span class="o">&lt;</span><span class="n">Tuple2</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">Long</span><span class="o">&gt;&gt;</span><span class="n">ratesHistoryData</span> <span class= [...]
-ratesHistoryData.add(Tuple2.of("USD", 102L));
-ratesHistoryData.add(Tuple2.of("EUR", 114L));
-ratesHistoryData.add(Tuple2.of("YEN", 1L));
-ratesHistoryData.add(Tuple2.of("EUR", 116L));
-ratesHistoryData.add(Tuple2.of("USD", 105L));
+<span class="n">ratesHistoryData</span><span class="o">.</span><span class="na">add</span><span class="o">(</span><span class="n">Tuple2</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="s">&quot;USD&quot;</span><span class="o"> [...]
+<span class="n">ratesHistoryData</span><span class="o">.</span><span class="na">add</span><span class="o">(</span><span class="n">Tuple2</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="s">&quot;EUR&quot;</span><span class="o"> [...]
+<span class="n">ratesHistoryData</span><span class="o">.</span><span class="na">add</span><span class="o">(</span><span class="n">Tuple2</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="s">&quot;YEN&quot;</span><span class="o"> [...]
+<span class="n">ratesHistoryData</span><span class="o">.</span><span class="na">add</span><span class="o">(</span><span class="n">Tuple2</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="s">&quot;EUR&quot;</span><span class="o"> [...]
+<span class="n">ratesHistoryData</span><span class="o">.</span><span class="na">add</span><span class="o">(</span><span class="n">Tuple2</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="s">&quot;USD&quot;</span><span class="o"> [...]
-// Create and register an example table using the sample data set.
-DataStream<Tuple2<String, Long>> ratesHistoryStream = env.fromCollection(ratesHistoryData);
+<span class="c1">// Create and register an example table using the sample data set.</span>
+<span class="n">DataStream</span><span class="o">&lt;</span><span class="n">Tuple2</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">Long</span><span class="o">&gt;&gt;</span> <span class="n">ratesHistoryStream</span> <spa [...]
-Table ratesHistory = tEnv.fromDataStream(ratesHistoryStream, "r_currency, r_rate, r_proctime.proctime");
+<span class="n">Table</span> <span class="n">ratesHistory</span> <span class="o">=</span> <span class="n">tEnv</span><span class="o">.</span><span class="na">fromDataStream</span><span class="o">(</span><span class="n">ratesHistoryStream</span><span class="o">,</span> <span class="s&qu [...]
-tEnv.registerTable("RatesHistory", ratesHistory);
+<span class="n">tEnv</span><span class="o">.</span><span class="na">registerTable</span><span class="o">(</span><span class="s">&quot;RatesHistory&quot;</span><span class="o">,</span> <span class="n">ratesHistory</span><span class="o">);</span>
-// Create and register the temporal table function "rates".
-// Define "r_proctime" as the versioning field and "r_currency" as the primary key.
-TemporalTableFunction rates = ratesHistory.createTemporalTableFunction("r_proctime", "r_currency");
+<span class="c1">// Create and register the temporal table function &quot;rates&quot;.</span>
+<span class="c1">// Define &quot;r_proctime&quot; as the versioning field and &quot;r_currency&quot; as the primary key.</span>
+<span class="n">TemporalTableFunction</span> <span class="n">rates</span> <span class="o">=</span> <span class="n">ratesHistory</span><span class="o">.</span><span class="na">createTemporalTableFunction</span><span class="o">(</span><span class="s">&quot;r_proctime&quot;</span><span class="o"&g [...]
-tEnv.registerFunction("Rates", rates);
+<span class="n">tEnv</span><span class="o">.</span><span class="na">registerFunction</span><span class="o">(</span><span class="s">&quot;Rates&quot;</span><span class="o">,</span> <span class="n">rates</span><span class="o">);</span>
-(...)
-```
+<span class="o">(...)</span></code></pre></div>
-What does this **Rates** function do, in practice? Imagine we would like to check what the conversion rates looked like at a given time — say, 11:00. We could simply do something like:
+<p>What does this <strong>Rates</strong> function do, in practice? Imagine we would like to check what the conversion rates looked like at a given time — say, 11:00. We could simply do something like:</p>
-```sql
-SELECT * FROM Rates('11:00');
-```
+<div class="highlight"><pre><code class="language-sql"><span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">Rates</span><span class="p">(</span><span class="s1">&#39;11:00&#39;</span><span class="p">);</span></code></pre></div>
<center>
-<img src="{{ site.baseurl }}/img/blog/2019-05-13-temporal-tables/TemporalTables2.png" width="650px" alt="Point-in-time Querying"/>
+<img src="/img/blog/2019-05-13-temporal-tables/TemporalTables2.png" width="650px" alt="Point-in-time Querying" />
</center>
-<br>
+<p><br /></p>
-Even though Flink does not yet support querying temporal table functions with a constant time attribute parameter, these functions can be used to cover a much more interesting scenario: temporal table joins.
+<p>Even though Flink does not yet support querying temporal table functions with a constant time attribute parameter, these functions can be used to cover a much more interesting scenario: temporal table joins.</p>
-## Streaming Joins using Temporal Tables
+<h2 id="streaming-joins-using-temporal-tables">Streaming Joins using Temporal Tables</h2>
-Temporal tables reach their full potential when used in combination — erm, joined — with streaming data, for instance to power applications that must continuously whitelist against a reference dataset that changes over time for auditing or regulatory compliance. While efficient joins have long been an enduring challenge for query processors due to computational cost and resource consumption, joins over streaming data carry some additional challenges:
+<p>Temporal tables reach their full potential when used in combination — erm, joined — with streaming data, for instance to power applications that must continuously whitelist against a reference dataset that changes over time for auditing or regulatory compliance. While efficient joins have long been an enduring challenge for query processors due to computational cost and resource consumption, joins over streaming data carry some additional challenges:</p>
-* The **unbounded** nature of streams means that inputs are continuously evaluated and intermediate join results can consume memory resources indefinitely. Flink gracefully manages its memory consumption out-of-the-box (even for heavier cases where joins require spilling to disk) and supports time-windowed joins to bound the amount of data that needs to be kept around as state;
-* Streaming data might be **out-of-order** and **late**, so it is not possible to enforce an ordering upfront and time handling requires some thinking to avoid unnecessary outputs and retractions.
+<ul>
+ <li>The <strong>unbounded</strong> nature of streams means that inputs are continuously evaluated and intermediate join results can consume memory resources indefinitely. Flink gracefully manages its memory consumption out-of-the-box (even for heavier cases where joins require spilling to disk) and supports time-windowed joins to bound the amount of data that needs to be kept around as state;</li>
+ <li>Streaming data might be <strong>out-of-order</strong> and <strong>late</strong>, so it is not possible to enforce an ordering upfront and time handling requires some thinking to avoid unnecessary outputs and retractions.</li>
+</ul>
-In the particular case of temporal data, time-windowed joins are not enough (well, at least not without getting into some expensive tweaking): sooner or later, each reference record will fall outside of the window and be wiped from state, no longer being considered for future join results. To address this limitation, Flink has introduced support for temporal table joins to cover time-varying relations.
+<p>In the particular case of temporal data, time-windowed joins are not enough (well, at least not without getting into some expensive tweaking): sooner or later, each reference record will fall outside of the window and be wiped from state, no longer being considered for future join results. To address this limitation, Flink has introduced support for temporal table joins to cover time-varying relations.</p>
<center>
-<img src="{{ site.baseurl }}/img/blog/2019-05-13-temporal-tables/TemporalTables3.png" width="500px" alt="Temporal Table Join between Taxi Fares and Conversion Rates"/>
+<img src="/img/blog/2019-05-13-temporal-tables/TemporalTables3.png" width="500px" alt="Temporal Table Join between Taxi Fares and Conversion Rates" />
</center>
-<br>
+<p><br /></p>
-Each record from the append-only table on the probe side (```Taxi Fare```) is joined with the version of the record from the temporal table on the build side (```Conversion Rate```) that most closely matches the probe side record time attribute (```time```) for the same value of the primary key (```currency```). Remember the temporal table function (```Rates```) we registered earlier? It can now be used to express this join as a simple SQL statement that would otherwise require a heavier [...]
+<p>Each record from the append-only table on the probe side (<code>Taxi Fare</code>) is joined with the version of the record from the temporal table on the build side (<code>Conversion Rate</code>) that most closely matches the probe side record time attribute (<code>time</code>) for the same value of the primary key (<code>currency</code>). Remember the temporal table function (<code>Rates</code>) we registered earlier? [...]
<center>
-<img src="{{ site.baseurl }}/img/blog/2019-05-13-temporal-tables/TemporalTables4.png" width="700px" alt="Regular Join vs. Temporal Table Join"/>
+<img src="/img/blog/2019-05-13-temporal-tables/TemporalTables4.png" width="700px" alt="Regular Join vs. Temporal Table Join" />
</center>
-<br>
+<p><br /></p>
-Temporal table joins support both [processing]({{ site.DOCS_BASE_URL }}flink-docs-release-1.8/dev/table/streaming/joins.html#processing-time-temporal-joins) and [event time]({{ site.DOCS_BASE_URL }}flink-docs-release-1.8/dev/table/streaming/joins.html#event-time-temporal-joins) semantics and effectively limit the amount of data kept in state while also allowing records on the build side to be arbitrarily old, as opposed to time-windowed joins. Probe-side records only need to be kept in s [...]
+<p>Temporal table joins support both <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev/table/streaming/joins.html#processing-time-temporal-joins">processing</a> and <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev/table/streaming/joins.html#event-time-temporal-joins">event time</a> semantics and effectively limit the amount of data kept in state while also allowing records on the build side t [...]
-* Narrowing the **scope** of the join: only the time-matching version of ```ratesHistory``` is visible for a given ```taxiFare.time```;
-* Pruning **unneeded records** from state: for cases using event time, records between current time and the [watermark]({{ site.DOCS_BASE_URL }}flink-docs-release-1.8/dev/event_time.html#event-time-and-watermarks) delay are persisted for both the probe and build side. These are discarded as soon as the watermark arrives and the results are emitted — allowing the join operation to move forward in time and the build table to “refresh” its version in state.
+<ul>
+ <li>Narrowing the <strong>scope</strong> of the join: only the time-matching version of <code>ratesHistory</code> is visible for a given <code>taxiFare.time</code>;</li>
+ <li>Pruning <strong>unneeded records</strong> from state: for cases using event time, records between current time and the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev/event_time.html#event-time-and-watermarks">watermark</a> delay are persisted for both the probe and build side. These are discarded as soon as the watermark arrives and the results are emitted — allowing the join operation to move forward in time and the [...]
+</ul>
-## Conclusion
+<h2 id="conclusion">Conclusion</h2>
-All this means it is now possible to express continuous stream enrichment in relational and time-varying terms using Flink without dabbling into syntactic patchwork or compromising performance. In other words: stream time-travelling minus the flux capacitor. Extending this syntax to batch processing for enriching historic data with proper (event) time semantics is also part of the Flink roadmap!
+<p>All this means it is now possible to express continuous stream enrichment in relational and time-varying terms using Flink without dabbling into syntactic patchwork or compromising performance. In other words: stream time-travelling minus the flux capacitor. Extending this syntax to batch processing for enriching historic data with proper (event) time semantics is also part of the Flink roadmap!</p>
-If you'd like to get some **hands-on practice in joining streams with Flink SQL** (and Flink SQL in general), checkout this [free training for Flink SQL](https://github.com/ververica/sql-training/wiki). The training environment is based on Docker and set up in just a few minutes.
+<p>If you’d like to get some <strong>hands-on practice in joining streams with Flink SQL</strong> (and Flink SQL in general), checkout this <a href="https://github.com/ververica/sql-training/wiki">free training for Flink SQL</a>. The training environment is based on Docker and set up in just a few minutes.</p>
-Subscribe to the [Apache Flink mailing lists]({{ site.baseurl }}/community.html#mailing-lists) to stay up-to-date with the latest developments in this space.
+<p>Subscribe to the <a href="/community.html#mailing-lists">Apache Flink mailing lists</a> to stay up-to-date with the latest developments in this space.</p>
</description>
-<pubDate>Tue, 14 May 2019 12:00:00 +0000</pubDate>
+<pubDate>Tue, 14 May 2019 14:00:00 +0200</pubDate>
<link>https://flink.apache.org/2019/05/14/temporal-tables.html</link>
<guid isPermaLink="true">/2019/05/14/temporal-tables.html</guid>
</item>
<item>
<title>When Flink & Pulsar Come Together</title>
-<description>The open source data technology frameworks [Apache Flink](https://flink.apache.org/) and [Apache Pulsar](https://pulsar.apache.org/en/) can integrate in different ways to provide elastic data processing at large scale. I recently gave a talk at [Flink Forward](https://www.flink-forward.org/) San Francisco 2019 and presented some of the integrations between the two frameworks for batch and streaming applications. In this post, I will give a short introduction to Apache Pulsar [...]
+<description><p>The open source data technology frameworks <a href="https://flink.apache.org/">Apache Flink</a> and <a href="https://pulsar.apache.org/en/">Apache Pulsar</a> can integrate in different ways to provide elastic data processing at large scale. I recently gave a talk at <a href="https://www.flink-forward.org/">Flink Forward</a> San Francisco 2019 and presented some of the integrations between the two fram [...]
-## A brief introduction to Apache Pulsar
+<h2 id="a-brief-introduction-to-apache-pulsar">A brief introduction to Apache Pulsar</h2>
-[Apache Pulsar](https://pulsar.apache.org/en/) is an open-source distributed pub-sub messaging system under the stewardship of the [Apache Software Foundation](https://www.apache.org/). Pulsar is a multi-tenant, high-performance solution for server-to-server messaging including multiple features such as native support for multiple clusters in a Pulsar instance, with seamless [geo-replication](https://pulsar.apache.org/docs/en/administration-geo) of messages across clusters, very low publ [...]
+<p><a href="https://pulsar.apache.org/en/">Apache Pulsar</a> is an open-source distributed pub-sub messaging system under the stewardship of the <a href="https://www.apache.org/">Apache Software Foundation</a>. Pulsar is a multi-tenant, high-performance solution for server-to-server messaging including multiple features such as native support for multiple clusters in a Pulsar instance, with seamless <a href="https://pulsar.apache. [...]
-The first differentiating factor stems from the fact that although Pulsar provides a flexible pub-sub messaging system it is also backed by durable log storage — hence combining both messaging and storage under one framework. Because of that layered architecture, Pulsar provides instant failure recovery, independent scalability and balance-free cluster expansion.
+<p>The first differentiating factor stems from the fact that although Pulsar provides a flexible pub-sub messaging system it is also backed by durable log storage — hence combining both messaging and storage under one framework. Because of that layered architecture, Pulsar provides instant failure recovery, independent scalability and balance-free cluster expansion.</p>
-Pulsar’s architecture follows a similar pattern to other pub-sub systems as the framework is organized in topics as the main data entity, with producers sending data to, and consumers receiving data from a topic as shown in the diagram below.
+<p>Pulsar’s architecture follows a similar pattern to other pub-sub systems as the framework is organized in topics as the main data entity, with producers sending data to, and consumers receiving data from a topic as shown in the diagram below.</p>
<center>
-<img src="{{ site.baseurl }}/img/blog/pulsar-flink/image-1.png" width="400px" alt="Pulsar producers and consumers"/>
+<img src="/img/blog/pulsar-flink/image-1.png" width="400px" alt="Pulsar producers and consumers" />
</center>
-<br>
+<p><br /></p>
-The second differentiator of Pulsar is that the framework is built from the get-go with [multi-tenancy](https://pulsar.apache.org/docs/en/concepts-multi-tenancy/) in mind. What that means is that each Pulsar topic has a hierarchical management structure making the allocation of resources as well as the resource management and coordination between teams efficient and easy. With Pulsar’s multi-tenancy structure, data platform maintainers can onboard new teams with no friction as Pulsar pro [...]
+<p>The second differentiator of Pulsar is that the framework is built from the get-go with <a href="https://pulsar.apache.org/docs/en/concepts-multi-tenancy/">multi-tenancy</a> in mind. What that means is that each Pulsar topic has a hierarchical management structure making the allocation of resources as well as the resource management and coordination between teams efficient and easy. With Pulsar’s multi-tenancy structure, data platform maintainers can onboar [...]
<center>
-<img src="{{ site.baseurl }}/img/blog/pulsar-flink/image-2.png" width="640px" alt="Apache Flink and Apache Pulsar"/>
+<img src="/img/blog/pulsar-flink/image-2.png" width="640px" alt="Apache Flink and Apache Pulsar" />
</center>
-<br>
+<p><br /></p>
-Finally, Pulsar’s flexible messaging framework unifies the streaming and queuing data consumption models and provides greater flexibility. As shown in the below diagram, Pulsar holds the data in the topic while multiple teams can consume the data independently depending on their workloads and data consumption patterns.
+<p>Finally, Pulsar’s flexible messaging framework unifies the streaming and queuing data consumption models and provides greater flexibility. As shown in the below diagram, Pulsar holds the data in the topic while multiple teams can consume the data independently depending on their workloads and data consumption patterns.</p>
<center>
-<img src="{{ site.baseurl }}/img/blog/pulsar-flink/image-3.png" width="640px" alt="Apache Flink and Apache Pulsar"/>
+<img src="/img/blog/pulsar-flink/image-3.png" width="640px" alt="Apache Flink and Apache Pulsar" />
</center>
-<br>
+<p><br /></p>
-## Pulsar’s view on data: Segmented data streams
+<h2 id="pulsars-view-on-data-segmented-data-streams">Pulsar’s view on data: Segmented data streams</h2>
-Apache Flink is a streaming-first computation framework that perceives [batch processing as a special case of streaming]({{ site.baseurl }}/news/2019/02/13/unified-batch-streaming-blink.html). Flink’s view on data streams distinguishes batch and stream processing between bounded and unbounded data streams, assuming that for batch workloads the data stream is finite, with a beginning and an end.
+<p>Apache Flink is a streaming-first computation framework that perceives <a href="/news/2019/02/13/unified-batch-streaming-blink.html">batch processing as a special case of streaming</a>. Flink’s view on data streams distinguishes batch and stream processing between bounded and unbounded data streams, assuming that for batch workloads the data stream is finite, with a beginning and an end.</p>
-Apache Pulsar has a similar perspective to that of Apache Flink with regards to the data layer. The framework also uses streams as a unified view on all data, while its layered architecture allows traditional pub-sub messaging for streaming workloads and continuous data processing or usage of *Segmented Streams* and bounded data stream for batch and static workloads.
+<p>Apache Pulsar has a similar perspective to that of Apache Flink with regards to the data layer. The framework also uses streams as a unified view on all data, while its layered architecture allows traditional pub-sub messaging for streaming workloads and continuous data processing or usage of <em>Segmented Streams</em> and bounded data stream for batch and static workloads.</p>
<center>
-<img src="{{ site.baseurl }}/img/blog/pulsar-flink/image-4.png" width="640px" alt="Apache Flink and Apache Pulsar"/>
+<img src="/img/blog/pulsar-flink/image-4.png" width="640px" alt="Apache Flink and Apache Pulsar" />
</center>
-<br>
+<p><br /></p>
-With Pulsar, once a producer sends data to a topic, it is partitioned depending on the data traffic and then further segmented under those partitions — using Apache Bookkeeper as segment store — to allow for parallel data processing as illustrated in the diagram below. This allows a combination of traditional pub-sub messaging and distributed parallel computations in one framework.
+<p>With Pulsar, once a producer sends data to a topic, it is partitioned depending on the data traffic and then further segmented under those partitions — using Apache Bookkeeper as segment store — to allow for parallel data processing as illustrated in the diagram below. This allows a combination of traditional pub-sub messaging and distributed parallel computations in one framework.</p>
<center>
-<img src="{{ site.baseurl }}/img/blog/pulsar-flink/image-5.png" width="640px" alt="Apache Flink and Apache Pulsar"/>
+<img src="/img/blog/pulsar-flink/image-5.png" width="640px" alt="Apache Flink and Apache Pulsar" />
</center>
-<br>
-
-## When Flink + Pulsar come together
-
-Apache Flink and Apache Pulsar integrate in multiple ways already. In the following sections, I will present some potential future integrations between the frameworks and share examples of existing ways in which you can utilize the frameworks together.
-
-### Potential Integrations
-
-Pulsar can integrate with Apache Flink in different ways. Some potential integrations include providing support for streaming workloads with the use of *Streaming Connectors* and support for batch workloads with the use of *Batch Source Connectors*. Pulsar also comes with native support for schema that can integrate with Flink and provide structured access to the data, for example by using Flink SQL as a way of querying data in Pulsar. Finally, an alternative way of integrating the techn [...]
-
-From an architecture point of view, we can imagine the integration between the two frameworks as one that uses Apache Pulsar for a unified view of the data layer and Apache Flink as a unified computation and data processing framework and API.
-
-
-### Existing Integrations
-
-Integration between the two frameworks is ongoing and developers can already use Pulsar with Flink in multiple ways. For example, Pulsar can be used as a streaming source and streaming sink in Flink DataStream applications. Developers can ingest data from Pulsar into a Flink job that makes computations and processes real-time data, to then send the data back to a Pulsar topic as a streaming sink. Such an example is shown below:
-
-
-```java
-// create and configure Pulsar consumer
-PulsarSourceBuilder<String>builder = PulsarSourceBuilder
- .builder(new SimpleStringSchema())
- .serviceUrl(serviceUrl)
- .topic(inputTopic)
- .subscriptionName(subscription);
-SourceFunction<String> src = builder.build();
-// ingest DataStream with Pulsar consumer
-DataStream<String> words = env.addSource(src);
-
-// perform computation on DataStream (here a simple WordCount)
-DataStream<WordWithCount> wc = words
- .flatMap((FlatMapFunction<String, WordWithCount>) (word, collector) -> {
- collector.collect(new WordWithCount(word, 1));
- })
- .returns(WordWithCount.class)
- .keyBy("word")
- .timeWindow(Time.seconds(5))
- .reduce((ReduceFunction<WordWithCount>) (c1, c2) ->
- new WordWithCount(c1.word, c1.count + c2.count));
-
-// emit result via Pulsar producer
-wc.addSink(new FlinkPulsarProducer<>(
- serviceUrl,
- outputTopic,
- new AuthenticationDisabled(),
- wordWithCount -> wordWithCount.toString().getBytes(UTF_8),
- wordWithCount -> wordWithCount.word)
-);
-```
-
-Another integration between the two frameworks that developers can take advantage of includes using Pulsar as both a streaming source and a streaming table sink for Flink SQL or Table API queries as shown in the example below:
-
-```java
-// obtain a DataStream with words
-DataStream<String> words = ...
-
-// register DataStream as Table "words" with two attributes ("word", "ts").
-// "ts" is an event-time timestamp.
-tableEnvironment.registerDataStream("words", words, "word, ts.rowtime");
-
-// create a TableSink that produces to Pulsar
-TableSink sink = new PulsarJsonTableSink(
- serviceUrl,
- outputTopic,
- new AuthenticationDisabled(),
- ROUTING_KEY);
-
-// register Pulsar TableSink as table "wc"
-tableEnvironment.registerTableSink(
- "wc",
- sink.configure(
- new String[]{"word", "cnt"},
- new TypeInformation[]{Types.STRING, Types.LONG}));
-
-// count words per 5 seconds and write result to table "wc"
-tableEnvironment.sqlUpdate(
- "INSERT INTO wc " +
- "SELECT word, COUNT(*) AS cnt " +
- "FROM words " +
- "GROUP BY word, TUMBLE(ts, INTERVAL '5' SECOND)");
-```
-
-Finally, Flink integrates with Pulsar for batch workloads as a batch sink where all results get pushed to Pulsar after Apache Flink has completed the computation in a static data set. Such an example is shown below:
-
-```java
-// obtain DataSet from arbitrary computation
-DataSet<WordWithCount> wc = ...
-
-// create PulsarOutputFormat instance
-OutputFormat pulsarOutputFormat = new PulsarOutputFormat(
- serviceUrl,
- topic,
- new AuthenticationDisabled(),
- wordWithCount -> wordWithCount.toString().getBytes());
-// write DataSet to Pulsar
-wc.output(pulsarOutputFormat);
-```
-
-## Conclusion
-
-Both Pulsar and Flink share a similar view on how the data and the computation level of an application can be *“streaming-first”* with batch as a special case streaming. With Pulsar’s Segmented Streams approach and Flink’s steps to unify batch and stream processing workloads under one framework, there are numerous ways of integrating the two technologies together to provide elastic data processing at massive scale. Subscribe to the [Apache Flink]({{ site.baseurl }}/community.html#mailing [...]
+<p><br /></p>
+
+<h2 id="when-flink--pulsar-come-together">When Flink + Pulsar come together</h2>
+
+<p>Apache Flink and Apache Pulsar integrate in multiple ways already. In the following sections, I will present some potential future integrations between the frameworks and share examples of existing ways in which you can utilize the frameworks together.</p>
+
+<h3 id="potential-integrations">Potential Integrations</h3>
+
+<p>Pulsar can integrate with Apache Flink in different ways. Some potential integrations include providing support for streaming workloads with the use of <em>Streaming Connectors</em> and support for batch workloads with the use of <em>Batch Source Connectors</em>. Pulsar also comes with native support for schema that can integrate with Flink and provide structured access to the data, for example by using Flink SQL as a way of querying data in Pulsar. Final [...]
+
+<p>From an architecture point of view, we can imagine the integration between the two frameworks as one that uses Apache Pulsar for a unified view of the data layer and Apache Flink as a unified computation and data processing framework and API.</p>
+
+<h3 id="existing-integrations">Existing Integrations</h3>
+
+<p>Integration between the two frameworks is ongoing and developers can already use Pulsar with Flink in multiple ways. For example, Pulsar can be used as a streaming source and streaming sink in Flink DataStream applications. Developers can ingest data from Pulsar into a Flink job that makes computations and processes real-time data, to then send the data back to a Pulsar topic as a streaming sink. Such an example is shown below:</p>
+
+<div class="highlight"><pre><code class="language-java"><span class="c1">// create and configure Pulsar consumer</span>
+<span class="n">PulsarSourceBuilder</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span><span class="n">builder</span> <span class="o">=</span> <span class="n">PulsarSourceBuilder</span>
+ <span class="o">.</span><span class="na">builder</span><span class="o">(</span><span class="k">new</span> <span class="nf">SimpleStringSchema</span><span class="o">())</span>
+ <span class="o">.</span><span class="na">serviceUrl</span><span class="o">(</span><span class="n">serviceUrl</span><span class="o">)</span>
+ <span class="o">.</span><span class="na">topic</span><span class="o">(</span><span class="n">inputTopic</span><span class="o">)</span>
+ <span class="o">.</span><span class="na">subscriptionName</span><span class="o">(</span><span class="n">subscription</span><span class="o">);</span>
+<span class="n">SourceFunction</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">src</span> <span class="o">=</span> <span class="n">builder</span><span class="o">.</span><span class="na">build</span><span class="o"&g [...]
+<span class="c1">// ingest DataStream with Pulsar consumer</span>
+<span class="n">DataStream</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">words</span> <span class="o">=</span> <span class="n">env</span><span class="o">.</span><span class="na">addSource</span><span class="o"> [...]
+
+<span class="c1">// perform computation on DataStream (here a simple WordCount)</span>
+<span class="n">DataStream</span><span class="o">&lt;</span><span class="n">WordWithCount</span><span class="o">&gt;</span> <span class="n">wc</span> <span class="o">=</span> <span class="n">words</span>
+ <span class="o">.</span><span class="na">flatMap</span><span class="o">((</span><span class="n">FlatMapFunction</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">WordWithCount</span><span class="o">&gt;)</span> <span class=&quo [...]
+ <span class="n">collector</span><span class="o">.</span><span class="na">collect</span><span class="o">(</span><span class="k">new</span> <span class="nf">WordWithCount</span><span class="o">(</span><span class="n">word</span><span class="o">,</span> <span class="mi">1&l [...]
+ <span class="o">})</span>
+ <span class="o">.</span><span class="na">returns</span><span class="o">(</span><span class="n">WordWithCount</span><span class="o">.</span><span class="na">class</span><span class="o">)</span>
+ <span class="o">.</span><span class="na">keyBy</span><span class="o">(</span><span class="s">&quot;word&quot;</span><span class="o">)</span>
+ <span class="o">.</span><span class="na">timeWindow</span><span class="o">(</span><span class="n">Time</span><span class="o">.</span><span class="na">seconds</span><span class="o">(</span><span class="mi">5</span><span class="o">))</span>
+ <span class="o">.</span><span class="na">reduce</span><span class="o">((</span><span class="n">ReduceFunction</span><span class="o">&lt;</span><span class="n">WordWithCount</span><span class="o">&gt;)</span> <span class="o">(</span><span class="n">c1</span><span class="o&quo [...]
+ <span class="k">new</span> <span class="nf">WordWithCount</span><span class="o">(</span><span class="n">c1</span><span class="o">.</span><span class="na">word</span><span class="o">,</span> <span class="n">c1</span><span class="o">.</span><span class="na">count</span& [...]
+
+<span class="c1">// emit result via Pulsar producer</span>
+<span class="n">wc</span><span class="o">.</span><span class="na">addSink</span><span class="o">(</span><span class="k">new</span> <span class="n">FlinkPulsarProducer</span><span class="o">&lt;&gt;(</span>
+ <span class="n">serviceUrl</span><span class="o">,</span>
+ <span class="n">outputTopic</span><span class="o">,</span>
+ <span class="k">new</span> <span class="nf">AuthenticationDisabled</span><span class="o">(),</span>
+ <span class="n">wordWithCount</span> <span class="o">-&gt;</span> <span class="n">wordWithCount</span><span class="o">.</span><span class="na">toString</span><span class="o">().</span><span class="na">getBytes</span><span class="o">(</span><span class="n">UTF_8</span><span class=&quo [...]
+ <span class="n">wordWithCount</span> <span class="o">-&gt;</span> <span class="n">wordWithCount</span><span class="o">.</span><span class="na">word</span><span class="o">)</span>
+<span class="o">);</span></code></pre></div>
+
+<p>Another integration between the two frameworks that developers can take advantage of includes using Pulsar as both a streaming source and a streaming table sink for Flink SQL or Table API queries as shown in the example below:</p>
+
+<div class="highlight"><pre><code class="language-java"><span class="c1">// obtain a DataStream with words</span>
+<span class="n">DataStream</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">words</span> <span class="o">=</span> <span class="o">...</span>
+
+<span class="c1">// register DataStream as Table &quot;words&quot; with two attributes (&quot;word&quot;, &quot;ts&quot;). </span>
+<span class="c1">// &quot;ts&quot; is an event-time timestamp.</span>
+<span class="n">tableEnvironment</span><span class="o">.</span><span class="na">registerDataStream</span><span class="o">(</span><span class="s">&quot;words&quot;</span><span class="o">,</span> <span class="n">words</span><span class="o">,</span> <span class="s">&quot;word, ts.rowtime&am [...]
+
+<span class="c1">// create a TableSink that produces to Pulsar</span>
+<span class="n">TableSink</span> <span class="n">sink</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">PulsarJsonTableSink</span><span class="o">(</span>
+ <span class="n">serviceUrl</span><span class="o">,</span>
+ <span class="n">outputTopic</span><span class="o">,</span>
+ <span class="k">new</span> <span class="nf">AuthenticationDisabled</span><span class="o">(),</span>
+ <span class="n">ROUTING_KEY</span><span class="o">);</span>
+
+<span class="c1">// register Pulsar TableSink as table &quot;wc&quot;</span>
+<span class="n">tableEnvironment</span><span class="o">.</span><span class="na">registerTableSink</span><span class="o">(</span>
+ <span class="s">&quot;wc&quot;</span><span class="o">,</span>
+ <span class="n">sink</span><span class="o">.</span><span class="na">configure</span><span class="o">(</span>
+ <span class="k">new</span> <span class="n">String</span><span class="o">[]{</span><span class="s">&quot;word&quot;</span><span class="o">,</span> <span class="s">&quot;cnt&quot;</span><span class="o">},</span>
+ <span class="k">new</span> <span class="n">TypeInformation</span><span class="o">[]{</span><span class="n">Types</span><span class="o">.</span><span class="na">STRING</span><span class="o">,</span> <span class="n">Types</span><span class="o">.</span><span class="na">LONG& [...]
+
+<span class="c1">// count words per 5 seconds and write result to table &quot;wc&quot;</span>
+<span class="n">tableEnvironment</span><span class="o">.</span><span class="na">sqlUpdate</span><span class="o">(</span>
+ <span class="s">&quot;INSERT INTO wc &quot;</span> <span class="o">+</span>
+ <span class="s">&quot;SELECT word, COUNT(*) AS cnt &quot;</span> <span class="o">+</span>
+ <span class="s">&quot;FROM words &quot;</span> <span class="o">+</span>
+ <span class="s">&quot;GROUP BY word, TUMBLE(ts, INTERVAL &#39;5&#39; SECOND)&quot;</span><span class="o">);</span></code></pre></div>
+
+<p>Finally, Flink integrates with Pulsar for batch workloads as a batch sink where all results get pushed to Pulsar after Apache Flink has completed the computation in a static data set. Such an example is shown below:</p>
+
+<div class="highlight"><pre><code class="language-java"><span class="c1">// obtain DataSet from arbitrary computation</span>
+<span class="n">DataSet</span><span class="o">&lt;</span><span class="n">WordWithCount</span><span class="o">&gt;</span> <span class="n">wc</span> <span class="o">=</span> <span class="o">...</span>
+
+<span class="c1">// create PulsarOutputFormat instance</span>
+<span class="n">OutputFormat</span> <span class="n">pulsarOutputFormat</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">PulsarOutputFormat</span><span class="o">(</span>
+ <span class="n">serviceUrl</span><span class="o">,</span>
+ <span class="n">topic</span><span class="o">,</span>
+ <span class="k">new</span> <span class="nf">AuthenticationDisabled</span><span class="o">(),</span>
+ <span class="n">wordWithCount</span> <span class="o">-&gt;</span> <span class="n">wordWithCount</span><span class="o">.</span><span class="na">toString</span><span class="o">().</span><span class="na">getBytes</span><span class="o">());</span>
+<span class="c1">// write DataSet to Pulsar</span>
+<span class="n">wc</span><span class="o">.</span><span class="na">output</span><span class="o">(</span><span class="n">pulsarOutputFormat</span><span class="o">);</span></code></pre></div>
+
+<h2 id="conclusion">Conclusion</h2>
+
+<p>Both Pulsar and Flink share a similar view on how the data and the computation level of an application can be <em>“streaming-first”</em> with batch as a special case streaming. With Pulsar’s Segmented Streams approach and Flink’s steps to unify batch and stream processing workloads under one framework, there are numerous ways of integrating the two technologies together to provide elastic data processing at massive scale. Subscribe to the <a href="/community. [...]
</description>
-<pubDate>Fri, 03 May 2019 12:00:00 +0000</pubDate>
+<pubDate>Fri, 03 May 2019 14:00:00 +0200</pubDate>
<link>https://flink.apache.org/2019/05/03/pulsar-flink.html</link>
<guid isPermaLink="true">/2019/05/03/pulsar-flink.html</guid>
</item>
<item>
<title>Apache Flink's Application to Season of Docs</title>
-<description>The Apache Flink community is happy to announce its application to the first edition of [Season of Docs](https://developers.google.com/season-of-docs/) by Google. The program is bringing together Open Source projects and technical writers to raise awareness for and improve documentation of Open Source projects. While the community is continuously looking for new contributors to collaborate on our documentation, we would like to take this chance to work with one or two techni [...]
+<description><p>The Apache Flink community is happy to announce its application to the first edition of <a href="https://developers.google.com/season-of-docs/">Season of Docs</a> by Google. The program is bringing together Open Source projects and technical writers to raise awareness for and improve documentation of Open Source projects. While the community is continuously looking for new contributors to collaborate on our documentation, we would like to take [...]
-The community has discussed this opportunity on the [dev mailinglist](https://lists.apache.org/thread.html/3c789b6187da23ad158df59bbc598543b652e3cfc1010a14e294e16a@%3Cdev.flink.apache.org%3E) and agreed on three project ideas to submit to the program. We have a great team of mentors (Stephan, Fabian, David, Jark & Konstantin) lined up and are very much looking forward to the first proposals by potential technical writers (given we are admitted to the program ;)). In case of questions [...]
+<p>The community has discussed this opportunity on the <a href="https://lists.apache.org/thread.html/3c789b6187da23ad158df59bbc598543b652e3cfc1010a14e294e16a@%3Cdev.flink.apache.org%3E">dev mailinglist</a> and agreed on three project ideas to submit to the program. We have a great team of mentors (Stephan, Fabian, David, Jark &amp; Konstantin) lined up and are very much looking forward to the first proposals by potential technical writers (given we are adm [...]
-## Project Ideas List
+<h2 id="project-ideas-list">Project Ideas List</h2>
-### Project 1: Improve Documentation of Stream Processing Concepts
+<h3 id="project-1-improve-documentation-of-stream-processing-concepts">Project 1: Improve Documentation of Stream Processing Concepts</h3>
-**Description:** Stream processing is the processing of data in motion―in other words, computing on data directly as it is produced or received. Apache Flink has pioneered the field of distributed, stateful stream processing over the last several years. As the community has pushed the boundaries of stream processing, we have introduced new concepts that users need to become familiar with to develop and operate Apache Flink applications efficiently.
-The Apache Flink documentation \[1\] already contains a “concepts” section, but it is a ) incomplete and b) lacks an overall structure & reading flow. In addition, “concepts”-content is also spread over the development \[2\] & operations \[3\] documentation without references to the “concepts” section. An example of this can be found in \[4\] and \[5\].
+<p><strong>Description:</strong> Stream processing is the processing of data in motion―in other words, computing on data directly as it is produced or received. Apache Flink has pioneered the field of distributed, stateful stream processing over the last several years. As the community has pushed the boundaries of stream processing, we have introduced new concepts that users need to become familiar with to develop and operate Apache Flink applications efficiently.
+The Apache Flink documentation [1] already contains a “concepts” section, but it is a ) incomplete and b) lacks an overall structure &amp; reading flow. In addition, “concepts”-content is also spread over the development [2] &amp; operations [3] documentation without references to the “concepts” section. An example of this can be found in [4] and [5].</p>
-In this project, we would like to restructure, consolidate and extend the concepts documentation for Apache Flink to better guide users who want to become productive as quickly as possible. This includes better conceptual introductions to topics such as event time, state, and fault tolerance with proper linking to and from relevant deployment and development guides.
+<p>In this project, we would like to restructure, consolidate and extend the concepts documentation for Apache Flink to better guide users who want to become productive as quickly as possible. This includes better conceptual introductions to topics such as event time, state, and fault tolerance with proper linking to and from relevant deployment and development guides.</p>
-**Related material:**
+<p><strong>Related material:</strong></p>
-1. [{{ site.DOCS_BASE_URL }}flink-docs-release-1.8/]({{ site.DOCS_BASE_URL }}flink-docs-release-1.8/)
-2. [{{ site.DOCS_BASE_URL }}flink-docs-release-1.8/dev]({{ site.DOCS_BASE_URL }}flink-docs-release-1.8/dev)
-3. [{{ site.DOCS_BASE_URL }}flink-docs-release-1.8/ops]({{ site.DOCS_BASE_URL }}flink-docs-release-1.8/ops)
-4. [{{ site.DOCS_BASE_URL }}flink-docs-release-1.8/concepts/programming-model.html#time]({{ site.DOCS_BASE_URL }}flink-docs-release-1.8/concepts/programming-model.html#time)
-5. [{{ site.DOCS_BASE_URL }}flink-docs-release-1.8/dev/event_time.html]({{ site.DOCS_BASE_URL }}flink-docs-release-1.8/dev/event_time.html)
+<ol>
+ <li><a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/">https://ci.apache.org/projects/flink/flink-docs-release-1.8/</a></li>
+ <li><a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev">https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev</a></li>
+ <li><a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/ops">https://ci.apache.org/projects/flink/flink-docs-release-1.8/ops</a></li>
+ <li><a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/concepts/programming-model.html#time">https://ci.apache.org/projects/flink/flink-docs-release-1.8/concepts/programming-model.html#time</a></li>
+ <li><a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev/event_time.html">https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev/event_time.html</a></li>
+</ol>
-### Project 2: Improve Documentation of Flink Deployments & Operations
+<h3 id="project-2-improve-documentation-of-flink-deployments--operations">Project 2: Improve Documentation of Flink Deployments &amp; Operations</h3>
-**Description:** Stream processing is the processing of data in motion―in other words, computing on data directly as it is produced or received. Apache Flink has pioneered the field of distributed, stateful stream processing for the last few years. As a stateful distributed system in general and a continuously running, low-latency system in particular, Apache Flink deployments are non-trivial to setup and manage.
-Unfortunately, the operations \[1\] and monitoring documentation \[2\] are arguably the weakest spots of the Apache Flink documentation. While it is comprehensive and often goes into a lot of detail, it lacks an overall structure and does not address common overarching concerns of operations teams in an efficient way.
+<p><strong>Description:</strong> Stream processing is the processing of data in motion―in other words, computing on data directly as it is produced or received. Apache Flink has pioneered the field of distributed, stateful stream processing for the last few years. As a stateful distributed system in general and a continuously running, low-latency system in particular, Apache Flink deployments are non-trivial to setup and manage.
+Unfortunately, the operations [1] and monitoring documentation [2] are arguably the weakest spots of the Apache Flink documentation. While it is comprehensive and often goes into a lot of detail, it lacks an overall structure and does not address common overarching concerns of operations teams in an efficient way.</p>
-In this project, we would like to restructure this part of the documentation and extend it if possible. Ideas for extension include: discussion of session and per-job clusters, better documentation for containerized deployments (incl. K8s), capacity planning & integration into CI/CD pipelines.
+<p>In this project, we would like to restructure this part of the documentation and extend it if possible. Ideas for extension include: discussion of session and per-job clusters, better documentation for containerized deployments (incl. K8s), capacity planning &amp; integration into CI/CD pipelines.</p>
-**Related material:**
+<p><strong>Related material:</strong></p>
-1. [{{ site.DOCS_BASE_URL }}flink-docs-release-1.8/ops]({{ site.DOCS_BASE_URL }}flink-docs-release-1.8/ops/)
-2. [{{ site.DOCS_BASE_URL }}flink-docs-release-1.8/monitoring]({{ site.DOCS_BASE_URL }}flink-docs-release-1.8/monitoring)
+<ol>
+ <li><a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/ops/">https://ci.apache.org/projects/flink/flink-docs-release-1.8/ops</a></li>
+ <li><a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/monitoring">https://ci.apache.org/projects/flink/flink-docs-release-1.8/monitoring</a></li>
+</ol>
-### Project 3: Improve Documentation for Relational APIs (Table API & SQL)
+<h3 id="project-3-improve-documentation-for-relational-apis-table-api--sql">Project 3: Improve Documentation for Relational APIs (Table API &amp; SQL)</h3>
-**Description:** Apache Flink features APIs at different levels of abstraction which enables its users to trade conciseness for expressiveness. Flink’s relational APIs, SQL and the Table API, are “younger” than the DataStream and DataSet APIs, more high-level and focus on data analytics use cases. A core principle of Flink’s SQL and Table API is that they can be used to process static (batch) and continuous (streaming) data and that a program or query produces the same result in both cases.
-The documentation of Flink’s relational APIs has organically grown and can be improved in a few areas. There are several on-going development efforts (e.g. Hive Integration, Python Support or Support for Interactive Programming) that aim to extend the scope of the Table API and SQL.
+<p><strong>Description:</strong> Apache Flink features APIs at different levels of abstraction which enables its users to trade conciseness for expressiveness. Flink’s relational APIs, SQL and the Table API, are “younger” than the DataStream and DataSet APIs, more high-level and focus on data analytics use cases. A core principle of Flink’s SQL and Table API is that they can be used to process static (batch) and continuous (streaming) data and that a program or query pr [...]
+The documentation of Flink’s relational APIs has organically grown and can be improved in a few areas. There are several on-going development efforts (e.g. Hive Integration, Python Support or Support for Interactive Programming) that aim to extend the scope of the Table API and SQL.</p>
-The existing documentation could be reorganized to prepare for covering the new features. Moreover, it could be improved by adding a concepts section that describes the use cases and internals of the APIs in more detail. Moreover, the documentation of built-in functions could be improved by adding more concrete examples.
+<p>The existing documentation could be reorganized to prepare for covering the new features. Moreover, it could be improved by adding a concepts section that describes the use cases and internals of the APIs in more detail. Moreover, the documentation of built-in functions could be improved by adding more concrete examples.</p>
-**Related material:**
+<p><strong>Related material:</strong></p>
-1. [Table API & SQL docs main page]({{ site.DOCS_BASE_URL }}flink-docs-release-1.8/dev/table)
-2. [Built-in functions]({{ site.DOCS_BASE_URL }}flink-docs-release-1.8/dev/table/functions.html)
-3. [Concepts]({{ site.DOCS_BASE_URL }}flink-docs-release-1.8/dev/table/common.html)
-4. [Streaming Concepts]({{ site.DOCS_BASE_URL }}flink-docs-release-1.8/dev/table/streaming/)
+<ol>
+ <li><a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev/table">Table API &amp; SQL docs main page</a></li>
+ <li><a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev/table/functions.html">Built-in functions</a></li>
+ <li><a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev/table/common.html">Concepts</a></li>
+ <li><a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev/table/streaming/">Streaming Concepts</a></li>
+</ol>
</description>
-<pubDate>Wed, 17 Apr 2019 12:00:00 +0000</pubDate>
+<pubDate>Wed, 17 Apr 2019 14:00:00 +0200</pubDate>
<link>https://flink.apache.org/news/2019/04/17/sod.html</link>
<guid isPermaLink="true">/news/2019/04/17/sod.html</guid>
</item>
<item>
<title>Apache Flink 1.8.0 Release Announcement</title>
-<description>The Apache Flink community is pleased to announce Apache Flink 1.8.0. The
+<description><p>The Apache Flink community is pleased to announce Apache Flink 1.8.0. The
latest release includes more than 420 resolved issues and some exciting
additions to Flink that we describe in the following sections of this post.
-Please check the [complete changelog](https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12344274)
-for more details.
+Please check the <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&amp;version=12344274">complete changelog</a>
+for more details.</p>
-Flink 1.8.0 is API-compatible with previous 1.x.y releases for APIs annotated
-with the `@Public` annotation. The release is available now and we encourage
-everyone to [download the release]({{ site.baseurl }}/downloads.html) and
+<p>Flink 1.8.0 is API-compatible with previous 1.x.y releases for APIs annotated
+with the <code>@Public</code> annotation. The release is available now and we encourage
+everyone to <a href="/downloads.html">download the release</a> and
check out the updated
-[documentation]({{ site.DOCS_BASE_URL }}flink-docs-release-1.8/).
-Feedback through the Flink [mailing
-lists]({{ site.baseurl }}/community.html#mailing-lists) or
-[JIRA](https://issues.apache.org/jira/projects/FLINK/summary) is, as always,
-very much appreciated!
-
-You can find the binaries on the updated [Downloads page]({{ site.baseurl }}/downloads.html) on the Flink project site.
+<a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/">documentation</a>.
+Feedback through the Flink <a href="/community.html#mailing-lists">mailing
+lists</a> or
+<a href="https://issues.apache.org/jira/projects/FLINK/summary">JIRA</a> is, as always,
+very much appreciated!</p>
+
+<p>You can find the binaries on the updated <a href="/downloads.html">Downloads page</a> on the Flink project site.</p>
+
+<div class="page-toc">
+<ul id="markdown-toc">
+ <li><a href="#new-features-and-improvements" id="markdown-toc-new-features-and-improvements">New Features and Improvements</a></li>
+ <li><a href="#important-changes" id="markdown-toc-important-changes">Important Changes</a></li>
+ <li><a href="#known-issues" id="markdown-toc-known-issues">Known Issues</a></li>
+ <li><a href="#release-notes" id="markdown-toc-release-notes">Release Notes</a></li>
+ <li><a href="#list-of-contributors" id="markdown-toc-list-of-contributors">List of Contributors</a></li>
+</ul>
-{% toc %}
+</div>
-With Flink 1.8.0 we come closer to our goals of enabling fast data processing
+<p>With Flink 1.8.0 we come closer to our goals of enabling fast data processing
and building data-intensive applications for the Flink community in a seamless
way. We do this by cleaning up and refactoring Flink under the hood to allow
more efficient feature development in the future. This includes removal of the
-legacy runtime components that were subsumed in the major rework of Flink's
+legacy runtime components that were subsumed in the major rework of Flink’s
underlying distributed system architecture
-([FLIP-6](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65147077))
+(<a href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65147077">FLIP-6</a>)
as well as refactorings on the Table API that prepare it for the future
addition of the Blink enhancements
-([FLINK-11439](https://issues.apache.org/jira/browse/FLINK-11439)).
+(<a href="https://issues.apache.org/jira/browse/FLINK-11439">FLINK-11439</a>).</p>
-Nevertheless, this release includes some important new features and bug fixes.
+<p>Nevertheless, this release includes some important new features and bug fixes.
The most interesting of those are highlighted below. Please consult the
-[complete changelog](https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12344274)
-and the [release notes]({{ site.DOCS_BASE_URL }}flink-docs-release-1.8/release-notes/flink-1.8.html)
-for more details.
-
-
-## New Features and Improvements
-
-* **Finalized State Schema Evolution Story**: This release completes
- the community driven effort to provide a schema evolution story for
- user state managed by Flink. This has been an effort that spanned 2
- releases, starting from 1.7.0 with the introduction of support for
- Avro state schema evolution as well as a revamped serialization
- compatibility abstraction.
-
- Flink 1.8.0 finalizes this effort by extending support for schema
- evolution to POJOs, upgrading all Flink built-in serializers to use
- the new serialization compatibility abstractions, as well as making it
- easier for advanced users who use custom state serializers to
- implement the abstractions. These different aspects for a complete
- out-of-the-box schema evolution story are explained in detail below:
-
- 1. Support for POJO state schema evolution: The pool of data types
- that support state schema evolution has been expanded to include
- POJOs. For state types that use POJOs, you can now add or remove
- fields from your POJO while retaining backwards
- compatibility. For a full overview of the list of data types that
- now support schema evolution as well as their evolution
- specifications and limitations, please refer to the State Schema
- Evolution documentation page.
-
-
- 2. Upgrade all Flink serializers to use new serialization
- compatibility asbtractions: Back in 1.7.0, we introduced the new
- serialization compatibility abstractions `TypeSerializerSnapshot`
- and `TypeSerializerSchemaCompatibility`. Besides providing a more
- expressible API to reflect schema compatibility between the data
- stored in savepoints and the data registered at runtime, another
- important aspect about the new abstraction is that it avoids the
- need for Flink to Java-serialize the state serializer as state
- metadata in savepoints.
-
- In 1.8.0, all of Flink's built-in serializers have been upgraded to
- use the new abstractions, and therefore the serializers
- themselves are no longer Java-serialized into savepoints. This
- greatly improves interoperability of Flink savepoints, in terms
- of state schema evolvability. For example, one outcome was the
- support for POJO schema evolution, as previously mentioned
- above. Another outcome is that all composite data types supported
- by Flink (such as `Either`, Scala case classes, Flink Java
- `Tuple`s, etc.) are generally evolve-able as well when they have
- a nested evolvable type, such as a POJO. For example, the `MyPojo`
- type in `ValueState<Tuple2<Integer, MyPojo>>` or
- `ListState<Either<Integer, MyPojo>>`, which is a POJO, is allowed
- to evolve its schema.
-
- For users who are using custom `TypeSerializer` implementations
- for their state serializer and are still using the outdated
- abstractions (i.e. `TypeSerializerConfigSnapshot` and
- `CompatiblityResult`), we highly recommend upgrading to the new
- abstractions to be future proof. Please refer to the Custom State
- Serialization documentation page for a detailed description on
- the new abstractions.
-
- 3. Provide pre-defined snapshot implementations for common
- serializers: For convenience, Flink 1.8.0 comes with two
- predefined implementations for the `TypeSerializerSnapshot` that
- make the task of implementing these new abstractions easier
- for most implementations of `TypeSerializer`s -
- `SimpleTypeSerializerSnapshot` and
- `CompositeTypeSerializerSnapshot`. This section in the
- documentation provides information on how to use these classes.
-
-* **Continuous cleanup of old state based on TTL
- ([FLINK-7811](https://issues.apache.org/jira/browse/FLINK-7811))**: We
- introduced TTL (time-to-live) for Keyed state in Flink 1.6
- ([FLINK-9510](https://issues.apache.org/jira/browse/FLINK-9510)). This
- feature enabled cleanup and made keyed state entries inaccessible after a
- defined timeout. In addition state would now also be cleaned up when
- writing a savepoint/checkpoint.
-
- Flink 1.8 introduces continuous cleanup of old entries for both the RocksDB
- state backend
- ([FLINK-10471](https://issues.apache.org/jira/browse/FLINK-10471)) and the heap
- state backend
- ([FLINK-10473](https://issues.apache.org/jira/browse/FLINK-10473)). This means
- that old entries (according to the TTL setting) are continuously cleaned up.
-
-* **SQL pattern detection with user-defined functions and
- aggregations**: The support of the MATCH_RECOGNIZE clause has been
- extended by multiple features. The addition of user-defined
- functions allows for custom logic during pattern detection
- ([FLINK-10597](https://issues.apache.org/jira/browse/FLINK-10597)),
- while adding aggregations allows for more complex CEP definitions,
- such as the following
- ([FLINK-7599](https://issues.apache.org/jira/browse/FLINK-7599)).
-
- ```
- SELECT *
- FROM Ticker
- MATCH_RECOGNIZE (
- ORDER BY rowtime
- MEASURES
- AVG(A.price) AS avgPrice
- ONE ROW PER MATCH
- AFTER MATCH SKIP TO FIRST B
- PATTERN (A+ B)
- DEFINE
- A AS AVG(A.price) < 15
- ) MR;
- ```
-
-* **RFC-compliant CSV format ([FLINK-9964](https://issues.apache.org/jira/browse/FLINK-9964))**: The SQL tables can now be read and written in
- an RFC-4180 standard compliant CSV table format. The format might also be
- useful for general DataStream API users.
-
-* **New KafkaDeserializationSchema that gives direct access to ConsumerRecord
- ([FLINK-8354](https://issues.apache.org/jira/browse/FLINK-8354))**: For the
- Flink `KafkaConsumers`, we introduced a new `KafkaDeserializationSchema` that
- gives direct access to the Kafka `ConsumerRecord`. This now allows access to
- all data that Kafka provides for a record, including the headers. This
- subsumes the `KeyedSerializationSchema` functionality, which is deprecated but
- still available for now.
-
-* **Per-shard watermarking option in FlinkKinesisConsumer
- ([FLINK-5697](https://issues.apache.org/jira/browse/FLINK-5697))**: The Kinesis
- Consumer can now emit periodic watermarks that are derived from per-shard watermarks,
- for correct event time processing with subtasks that consume multiple Kinesis shards.
-
-* **New consumer for DynamoDB Streams to capture table changes
- ([FLINK-4582](https://issues.apache.org/jira/browse/FLINK-4582))**: `FlinkDynamoDBStreamsConsumer`
- is a variant of the Kinesis consumer that supports retrieval of CDC-like streams from DynamoDB tables.
-
-* **Support for global aggregates for subtask coordination
- ([FLINK-10887](https://issues.apache.org/jira/browse/FLINK-10887))**:
- Designed as a solution for global source watermark tracking, `GlobalAggregateManager`
- allows sharing of information between parallel subtasks. This feature will
- be integrated into streaming connectors for watermark synchronization and
- can be used for other purposes with a user defined aggregator.
-
-## Important Changes
-
-* **Changes to bundling of Hadoop libraries with Flink
- ([FLINK-11266](https://issues.apache.org/jira/browse/FLINK-11266))**:
- Convenience binaries that include hadoop are no longer released.
-
- If a deployment relies on `flink-shaded-hadoop2` being included in
- `flink-dist`, then you must manually download a pre-packaged Hadoop
- jar from the optional components section of the [download
- page]({{ site.baseurl }}/downloads.html) and copy it into the
- `/lib` directory. Alternatively, a Flink distribution that includes
- hadoop can be built by packaging `flink-dist` and activating the
- `include-hadoop` maven profile.
-
- As hadoop is no longer included in `flink-dist` by default, specifying
- `-DwithoutHadoop` when packaging `flink-dist` no longer impacts the build.
-
-* **FlinkKafkaConsumer will now filter restored partitions based on topic
- specification
- ([FLINK-10342](https://issues.apache.org/jira/browse/FLINK-10342))**:
- Starting from Flink 1.8.0, the `FlinkKafkaConsumer` now always filters out
- restored partitions that are no longer associated with a specified topic to
- subscribe to in the restored execution. This behaviour did not exist in
- previous versions of the `FlinkKafkaConsumer`. If you wish to retain the
- previous behaviour, please use the
- `disableFilterRestoredPartitionsWithSubscribedTopics()` configuration method
- on the `FlinkKafkaConsumer`.
-
- Consider this example: if you had a Kafka Consumer that was consuming from
- topic `A`, you did a savepoint, then changed your Kafka consumer to instead
- consume from topic `B`, and then restarted your job from the savepoint.
- Before this change, your consumer would now consume from both topic `A` and
- `B` because it was stored in state that the consumer was consuming from topic
- `A`. With the change, your consumer would only consume from topic `B` after
- restore because it now filters the topics that are stored in state using the
- configured topics.
-
- * **Change in the Maven modules of Table API
- ([FLINK-11064](https://issues.apache.org/jira/browse/FLINK-11064))**: Users
- that had a `flink-table` dependency before, need to update their
- dependencies to `flink-table-planner` and the correct dependency of
- `flink-table-api-*`, depending on whether Java or Scala is used: one of
- `flink-table-api-java-bridge` or `flink-table-api-scala-bridge`.
-
-## Known Issues
-
-* **Discarded checkpoint can cause Tasks to fail
- ([FLINK-11662](https://issues.apache.org/jira/browse/FLINK-11662))**: There is
- a race condition that can lead to erroneous checkpoint failures. This mostly
- occurs when restarting from a savepoint or checkpoint takes a long time at the
- sources of a job. If you see random checkpointing failures that don't seem to
- have a good explanation you might be affected. Please see the Jira issue for
- more details and a workaround for the problem.
-
-
-## Release Notes
-
-Please review the [release
-notes]({{ site.DOCS_BASE_URL }}flink-docs-release-1.8/release-notes/flink-1.8.html)
+<a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&amp;version=12344274">complete changelog</a>
+and the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/release-notes/flink-1.8.html">release notes</a>
+for more details.</p>
+
+<h2 id="new-features-and-improvements">New Features and Improvements</h2>
+
+<ul>
+ <li>
+ <p><strong>Finalized State Schema Evolution Story</strong>: This release completes
+the community driven effort to provide a schema evolution story for
+user state managed by Flink. This has been an effort that spanned 2
+releases, starting from 1.7.0 with the introduction of support for
+Avro state schema evolution as well as a revamped serialization
+compatibility abstraction.</p>
+
+ <p>Flink 1.8.0 finalizes this effort by extending support for schema
+evolution to POJOs, upgrading all Flink built-in serializers to use
+the new serialization compatibility abstractions, as well as making it
+easier for advanced users who use custom state serializers to
+implement the abstractions. These different aspects for a complete
+out-of-the-box schema evolution story are explained in detail below:</p>
+
+ <ol>
+ <li>
+ <p>Support for POJO state schema evolution: The pool of data types
+that support state schema evolution has been expanded to include
+POJOs. For state types that use POJOs, you can now add or remove
+fields from your POJO while retaining backwards
+compatibility. For a full overview of the list of data types that
+now support schema evolution as well as their evolution
+specifications and limitations, please refer to the State Schema
+Evolution documentation page.</p>
+ </li>
+ <li>
+ <p>Upgrade all Flink serializers to use new serialization
+compatibility asbtractions: Back in 1.7.0, we introduced the new
+serialization compatibility abstractions <code>TypeSerializerSnapshot</code>
+and <code>TypeSerializerSchemaCompatibility</code>. Besides providing a more
+expressible API to reflect schema compatibility between the data
+stored in savepoints and the data registered at runtime, another
+important aspect about the new abstraction is that it avoids the
+need for Flink to Java-serialize the state serializer as state
+metadata in savepoints.</p>
+
+ <p>In 1.8.0, all of Flink’s built-in serializers have been upgraded to
+use the new abstractions, and therefore the serializers
+themselves are no longer Java-serialized into savepoints. This
+greatly improves interoperability of Flink savepoints, in terms
+of state schema evolvability. For example, one outcome was the
+support for POJO schema evolution, as previously mentioned
+above. Another outcome is that all composite data types supported
+by Flink (such as <code>Either</code>, Scala case classes, Flink Java
+<code>Tuple</code>s, etc.) are generally evolve-able as well when they have
+a nested evolvable type, such as a POJO. For example, the <code>MyPojo</code>
+type in <code>ValueState&lt;Tuple2&lt;Integer, MyPojo&gt;&gt;</code> or
+<code>ListState&lt;Either&lt;Integer, MyPojo&gt;&gt;</code>, which is a POJO, is allowed
+to evolve its schema.</p>
+
+ <p>For users who are using custom <code>TypeSerializer</code> implementations
+for their state serializer and are still using the outdated
+abstractions (i.e. <code>TypeSerializerConfigSnapshot</code> and
+<code>CompatiblityResult</code>), we highly recommend upgrading to the new
+abstractions to be future proof. Please refer to the Custom State
+Serialization documentation page for a detailed description on
+the new abstractions.</p>
+ </li>
+ <li>
+ <p>Provide pre-defined snapshot implementations for common
+serializers: For convenience, Flink 1.8.0 comes with two
+predefined implementations for the <code>TypeSerializerSnapshot</code> that
+make the task of implementing these new abstractions easier
+for most implementations of <code>TypeSerializer</code>s -
+<code>SimpleTypeSerializerSnapshot</code> and
+<code>CompositeTypeSerializerSnapshot</code>. This section in the
+documentation provides information on how to use these classes.</p>
+ </li>
+ </ol>
+ </li>
+ <li>
+ <p><strong>Continuous cleanup of old state based on TTL
+(<a href="https://issues.apache.org/jira/browse/FLINK-7811">FLINK-7811</a>)</strong>: We
+introduced TTL (time-to-live) for Keyed state in Flink 1.6
+(<a href="https://issues.apache.org/jira/browse/FLINK-9510">FLINK-9510</a>). This
+feature enabled cleanup and made keyed state entries inaccessible after a
+defined timeout. In addition state would now also be cleaned up when
+writing a savepoint/checkpoint.</p>
+
+ <p>Flink 1.8 introduces continuous cleanup of old entries for both the RocksDB
+state backend
+(<a href="https://issues.apache.org/jira/browse/FLINK-10471">FLINK-10471</a>) and the heap
+state backend
+(<a href="https://issues.apache.org/jira/browse/FLINK-10473">FLINK-10473</a>). This means
+that old entries (according to the TTL setting) are continuously cleaned up.</p>
+ </li>
+ <li>
+ <p><strong>SQL pattern detection with user-defined functions and
+aggregations</strong>: The support of the MATCH_RECOGNIZE clause has been
+extended by multiple features. The addition of user-defined
+functions allows for custom logic during pattern detection
+(<a href="https://issues.apache.org/jira/browse/FLINK-10597">FLINK-10597</a>),
+while adding aggregations allows for more complex CEP definitions,
+such as the following
+(<a href="https://issues.apache.org/jira/browse/FLINK-7599">FLINK-7599</a>).</p>
+
+ <div class="highlight"><pre><code>SELECT *
+FROM Ticker
+ MATCH_RECOGNIZE (
+ ORDER BY rowtime
+ MEASURES
+ AVG(A.price) AS avgPrice
+ ONE ROW PER MATCH
+ AFTER MATCH SKIP TO FIRST B
+ PATTERN (A+ B)
+ DEFINE
+ A AS AVG(A.price) &lt; 15
+ ) MR;
+</code></pre></div>
+ </li>
+ <li>
+ <p><strong>RFC-compliant CSV format (<a href="https://issues.apache.org/jira/browse/FLINK-9964">FLINK-9964</a>)</strong>: The SQL tables can now be read and written in
+an RFC-4180 standard compliant CSV table format. The format might also be
+useful for general DataStream API users.</p>
+ </li>
+ <li>
+ <p><strong>New KafkaDeserializationSchema that gives direct access to ConsumerRecord
+(<a href="https://issues.apache.org/jira/browse/FLINK-8354">FLINK-8354</a>)</strong>: For the
+Flink <code>KafkaConsumers</code>, we introduced a new <code>KafkaDeserializationSchema</code> that
+gives direct access to the Kafka <code>ConsumerRecord</code>. This now allows access to
+all data that Kafka provides for a record, including the headers. This
+subsumes the <code>KeyedSerializationSchema</code> functionality, which is deprecated but
+still available for now.</p>
+ </li>
+ <li>
+ <p><strong>Per-shard watermarking option in FlinkKinesisConsumer
+(<a href="https://issues.apache.org/jira/browse/FLINK-5697">FLINK-5697</a>)</strong>: The Kinesis
+Consumer can now emit periodic watermarks that are derived from per-shard watermarks,
+for correct event time processing with subtasks that consume multiple Kinesis shards.</p>
+ </li>
+ <li>
+ <p><strong>New consumer for DynamoDB Streams to capture table changes
+(<a href="https://issues.apache.org/jira/browse/FLINK-4582">FLINK-4582</a>)</strong>: <code>FlinkDynamoDBStreamsConsumer</code>
+is a variant of the Kinesis consumer that supports retrieval of CDC-like streams from DynamoDB tables.</p>
+ </li>
+ <li>
+ <p><strong>Support for global aggregates for subtask coordination
+(<a href="https://issues.apache.org/jira/browse/FLINK-10887">FLINK-10887</a>)</strong>:
+Designed as a solution for global source watermark tracking, <code>GlobalAggregateManager</code>
+allows sharing of information between parallel subtasks. This feature will
+be integrated into streaming connectors for watermark synchronization and
+can be used for other purposes with a user defined aggregator.</p>
+ </li>
+</ul>
+
+<h2 id="important-changes">Important Changes</h2>
+
+<ul>
+ <li>
+ <p><strong>Changes to bundling of Hadoop libraries with Flink
+(<a href="https://issues.apache.org/jira/browse/FLINK-11266">FLINK-11266</a>)</strong>:
+Convenience binaries that include hadoop are no longer released.</p>
+
+ <p>If a deployment relies on <code>flink-shaded-hadoop2</code> being included in
+<code>flink-dist</code>, then you must manually download a pre-packaged Hadoop
+jar from the optional components section of the <a href="/downloads.html">download
+page</a> and copy it into the
+<code>/lib</code> directory. Alternatively, a Flink distribution that includes
+hadoop can be built by packaging <code>flink-dist</code> and activating the
+<code>include-hadoop</code> maven profile.</p>
+
+ <p>As hadoop is no longer included in <code>flink-dist</code> by default, specifying
+<code>-DwithoutHadoop</code> when packaging <code>flink-dist</code> no longer impacts the build.</p>
+ </li>
+ <li>
+ <p><strong>FlinkKafkaConsumer will now filter restored partitions based on topic
+specification
+(<a href="https://issues.apache.org/jira/browse/FLINK-10342">FLINK-10342</a>)</strong>:
+Starting from Flink 1.8.0, the <code>FlinkKafkaConsumer</code> now always filters out
+restored partitions that are no longer associated with a specified topic to
+subscribe to in the restored execution. This behaviour did not exist in
+previous versions of the <code>FlinkKafkaConsumer</code>. If you wish to retain the
+previous behaviour, please use the
+<code>disableFilterRestoredPartitionsWithSubscribedTopics()</code> configuration method
+on the <code>FlinkKafkaConsumer</code>.</p>
+
+ <p>Consider this example: if you had a Kafka Consumer that was consuming from
+topic <code>A</code>, you did a savepoint, then changed your Kafka consumer to instead
+consume from topic <code>B</code>, and then restarted your job from the savepoint.
+Before this change, your consumer would now consume from both topic <code>A</code> and
+<code>B</code> because it was stored in state that the consumer was consuming from topic
+<code>A</code>. With the change, your consumer would only consume from topic <code>B</code> after
+restore because it now filters the topics that are stored in state using the
+configured topics.</p>
+ </li>
+ <li>
+ <p><strong>Change in the Maven modules of Table API
+(<a href="https://issues.apache.org/jira/browse/FLINK-11064">FLINK-11064</a>)</strong>: Users
+that had a <code>flink-table</code> dependency before, need to update their
+dependencies to <code>flink-table-planner</code> and the correct dependency of
+<code>flink-table-api-*</code>, depending on whether Java or Scala is used: one of
+<code>flink-table-api-java-bridge</code> or <code>flink-table-api-scala-bridge</code>.</p>
+ </li>
+</ul>
+
+<h2 id="known-issues">Known Issues</h2>
+
+<ul>
+ <li><strong>Discarded checkpoint can cause Tasks to fail
+(<a href="https://issues.apache.org/jira/browse/FLINK-11662">FLINK-11662</a>)</strong>: There is
+a race condition that can lead to erroneous checkpoint failures. This mostly
+occurs when restarting from a savepoint or checkpoint takes a long time at the
+sources of a job. If you see random checkpointing failures that don’t seem to
+have a good explanation you might be affected. Please see the Jira issue for
+more details and a workaround for the problem.</li>
+</ul>
+
+<h2 id="release-notes">Release Notes</h2>
+
+<p>Please review the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/release-notes/flink-1.8.html">release
+notes</a>
for a more detailed list of changes and new features if you plan to upgrade
-your Flink setup to Flink 1.8.
+your Flink setup to Flink 1.8.</p>
-## List of Contributors
+<h2 id="list-of-contributors">List of Contributors</h2>
-We would like to acknowledge all community members for contributing to this
+<p>We would like to acknowledge all community members for contributing to this
release. Special credits go to the following members for contributing to the
-1.8.0 release (according to `git log --pretty="%an" release-1.7.0..release-1.8.0 | sort | uniq` without manual deduplication):
+1.8.0 release (according to <code>git log --pretty="%an" release-1.7.0..release-1.8.0 | sort | uniq</code> without manual deduplication):</p>
-Addison Higham, Aitozi, Aleksey Pak, Alexander Fedulov, Alexey Trenikhin, Aljoscha Krettek, Andrey Zagrebin, Artsem Semianenka, Asura7969, Avi, Barisa Obradovic, Benchao Li, Bo WANG, Chesnay Schepler, Congxian Qiu, Cristian, David Anderson, Dawid Wysakowicz, Dian Fu, DuBin, EAlexRojas, EronWright, Eugen Yushin, Fabian Hueske, Fokko Driesprong, Gary Yao, Hequn Cheng, Igal Shilman, Jamie Grier, JaryZhen, Jeff Zhang, Jihyun Cho, Jinhu Wu, Joerg Schad, KarmaGYZ, Kezhu Wang, Konstantin Knauf, [...]
+<p>Addison Higham, Aitozi, Aleksey Pak, Alexander Fedulov, Alexey Trenikhin, Aljoscha Krettek, Andrey Zagrebin, Artsem Semianenka, Asura7969, Avi, Barisa Obradovic, Benchao Li, Bo WANG, Chesnay Schepler, Congxian Qiu, Cristian, David Anderson, Dawid Wysakowicz, Dian Fu, DuBin, EAlexRojas, EronWright, Eugen Yushin, Fabian Hueske, Fokko Driesprong, Gary Yao, Hequn Cheng, Igal Shilman, Jamie Grier, JaryZhen, Jeff Zhang, Jihyun Cho, Jinhu Wu, Joerg Schad, KarmaGYZ, Kezhu Wang, Konstant [...]
</description>
-<pubDate>Tue, 09 Apr 2019 12:00:00 +0000</pubDate>
+<pubDate>Tue, 09 Apr 2019 14:00:00 +0200</pubDate>
<link>https://flink.apache.org/news/2019/04/09/release-1.8.0.html</link>
<guid isPermaLink="true">/news/2019/04/09/release-1.8.0.html</guid>
</item>
<item>
<title>Flink and Prometheus: Cloud-native monitoring of streaming applications</title>
-<description>This blog post describes how developers can leverage Apache Flink's built-in [metrics system]({{ site.DOCS_BASE_URL }}flink-docs-release-1.7/monitoring/metrics.html) together with [Prometheus](https://prometheus.io/) to observe and monitor streaming applications in an effective way. This is a follow-up post from my [Flink Forward](https://flink-forward.org/) Berlin 2018 talk ([slides](https://www.slideshare.net/MaximilianBode1/monitoring-flink-with-prometheus), [video](h [...]
+<description><p>This blog post describes how developers can leverage Apache Flink’s built-in <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.7/monitoring/metrics.html">metrics system</a> together with <a href="https://prometheus.io/">Prometheus</a> to observe and monitor streaming applications in an effective way. This is a follow-up post from my <a href="https://flink-forward.org/">Flink Forward</a&g [...]
-## Why Prometheus?
+<h2 id="why-prometheus">Why Prometheus?</h2>
-Prometheus is a metrics-based monitoring system that was originally created in 2012. The system is completely open-source (under the Apache License 2) with a vibrant community behind it and it has graduated from the Cloud Native Foundation last year – a sign of maturity, stability and production-readiness. As we mentioned, the system is based on metrics and it is designed to measure the overall health, behavior and performance of a service. Prometheus features a multi-dimensional data mo [...]
+<p>Prometheus is a metrics-based monitoring system that was originally created in 2012. The system is completely open-source (under the Apache License 2) with a vibrant community behind it and it has graduated from the Cloud Native Foundation last year – a sign of maturity, stability and production-readiness. As we mentioned, the system is based on metrics and it is designed to measure the overall health, behavior and performance of a service. Prometheus features a multi-dimensiona [...]
-* **Metrics:** Prometheus defines metrics as floats of information that change in time. These time series have millisecond precision.
-
-* **Labels** are the key-value pairs associated with time series that support Prometheus' flexible and powerful data model – in contrast to hierarchical data structures that one might experience with traditional metrics systems.
-
-* **Scrape:** Prometheus is a pull-based system and fetches ("scrapes") metrics data from specified sources that expose HTTP endpoints with a text-based format.
-
-* **PromQL** is Prometheus' [query language](https://prometheus.io/docs/prometheus/latest/querying/basics/). It can be used for both building dashboards and setting up alert rules that will trigger when specific conditions are met.
+<ul>
+ <li>
+ <p><strong>Metrics:</strong> Prometheus defines metrics as floats of information that change in time. These time series have millisecond precision.</p>
+ </li>
+ <li>
+ <p><strong>Labels</strong> are the key-value pairs associated with time series that support Prometheus’ flexible and powerful data model – in contrast to hierarchical data structures that one might experience with traditional metrics systems.</p>
+ </li>
+ <li>
+ <p><strong>Scrape:</strong> Prometheus is a pull-based system and fetches (“scrapes”) metrics data from specified sources that expose HTTP endpoints with a text-based format.</p>
+ </li>
+ <li>
+ <p><strong>PromQL</strong> is Prometheus’ <a href="https://prometheus.io/docs/prometheus/latest/querying/basics/">query language</a>. It can be used for both building dashboards and setting up alert rules that will trigger when specific conditions are met.</p>
+ </li>
+</ul>
-When considering metrics and monitoring systems for your Flink jobs, there are many [options]({{ site.DOCS_BASE_URL }}flink-docs-release-1.7/monitoring/metrics.html). Flink offers native support for exposing data to Prometheus via the `PrometheusReporter` configuration. Setting up this integration is very easy.
+<p>When considering metrics and monitoring systems for your Flink jobs, there are many <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.7/monitoring/metrics.html">options</a>. Flink offers native support for exposing data to Prometheus via the <code>PrometheusReporter</code> configuration. Setting up this integration is very easy.</p>
-Prometheus is a great choice as usually Flink jobs are not running in isolation but in a greater context of microservices. For making metrics available to Prometheus from other parts of a larger system, there are two options: There exist [libraries for all major languages](https://prometheus.io/docs/instrumenting/clientlibs/) to instrument other applications. Additionally, there is a wide variety of [exporters](https://prometheus.io/docs/instrumenting/exporters/), which are tools that ex [...]
+<p>Prometheus is a great choice as usually Flink jobs are not running in isolation but in a greater context of microservices. For making metrics available to Prometheus from other parts of a larger system, there are two options: There exist <a href="https://prometheus.io/docs/instrumenting/clientlibs/">libraries for all major languages</a> to instrument other applications. Additionally, there is a wide variety of <a href="https://prometheus.io/docs/ins [...]
-## Prometheus and Flink in Action
+<h2 id="prometheus-and-flink-in-action">Prometheus and Flink in Action</h2>
-We have provided a [GitHub repository](https://github.com/mbode/flink-prometheus-example) that demonstrates the integration described above. To have a look, clone the repository, make sure [Docker](https://docs.docker.com/install/) is installed and run:
+<p>We have provided a <a href="https://github.com/mbode/flink-prometheus-example">GitHub repository</a> that demonstrates the integration described above. To have a look, clone the repository, make sure <a href="https://docs.docker.com/install/">Docker</a> is installed and run:</p>
-```
-./gradlew composeUp
-```
+<div class="highlight"><pre><code>./gradlew composeUp
+</code></pre></div>
-This builds a Flink job using the build tool [Gradle](https://gradle.org/) and starts up a local environment based on [Docker Compose](https://docs.docker.com/compose/) running the job in a [Flink job cluster]({{ site.DOCS_BASE_URL }}flink-docs-release-1.7/ops/deployment/docker.html#flink-job-cluster) (reachable at [http://localhost:8081](http://localhost:8081/)) as well as a Prometheus instance ([http://localhost:9090](http://localhost:9090/)).
+<p>This builds a Flink job using the build tool <a href="https://gradle.org/">Gradle</a> and starts up a local environment based on <a href="https://docs.docker.com/compose/">Docker Compose</a> running the job in a <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.7/ops/deployment/docker.html#flink-job-cluster">Flink job cluster</a> (reachable at <a href="http://localhost:8081/">http: [...]
<center>
-<img src="{{ site.baseurl }}/img/blog/2019-03-11-prometheus-monitoring/prometheusexamplejob.png" width="600px" alt="PrometheusExampleJob in Flink Web UI"/>
-<br/>
+<img src="/img/blog/2019-03-11-prometheus-monitoring/prometheusexamplejob.png" width="600px" alt="PrometheusExampleJob in Flink Web UI" />
+<br />
<i><small>Job graph and custom metric for example job in Flink web interface.</small></i>
</center>
-<br/>
+<p><br /></p>
-The `PrometheusExampleJob` has three operators: Random numbers up to 10,000 are generated, then a map counts the events and creates a histogram of the values passed through. Finally, the events are discarded without further output. The very simple code below is from the second operator. It illustrates how easy it is to add custom metrics relevant to your business logic into your Flink job.
+<p>The <code>PrometheusExampleJob</code> has three operators: Random numbers up to 10,000 are generated, then a map counts the events and creates a histogram of the values passed through. Finally, the events are discarded without further output. The very simple code below is from the second operator. It illustrates how easy it is to add custom metrics relevant to your business logic into your Flink job.</p>
-```java
-class FlinkMetricsExposingMapFunction extends RichMapFunction<Integer, Integer> {
- private transient Counter eventCounter;
+<div class="highlight"><pre><code class="language-java"><span class="kd">class</span> <span class="nc">FlinkMetricsExposingMapFunction</span> <span class="kd">extends</span> <span class="n">RichMapFunction</span><span class="o">&lt;</span><span class="n">Integer</span><span class="o">,</span> &l [...]
+ <span class="kd">private</span> <span class="kd">transient</span> <span class="n">Counter</span> <span class="n">eventCounter</span><span class="o">;</span>
- @Override
- public void open(Configuration parameters) {
- eventCounter = getRuntimeContext().getMetricGroup().counter("events");
- }
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="nf">open</span><span class="o">(</span><span class="n">Configuration</span> <span class="n">parameters</span><span class="o">)</span> <span class="o">{</span>
+ <span class="n">eventCounter</span> <span class="o">=</span> <span class="n">getRuntimeContext</span><span class="o">().</span><span class="na">getMetricGroup</span><span class="o">().</span><span class="na">counter</span><span class="o">(</span><span class="s">&quot;events&quot;</spa [...]
+ <span class="o">}</span>
- @Override
- public Integer map(Integer value) {
- eventCounter.inc();
- return value;
- }
-}
-```
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="n">Integer</span> <span class="nf">map</span><span class="o">(</span><span class="n">Integer</span> <span class="n">value</span><span class="o">)</span> <span class="o">{</span>
+ <span class="n">eventCounter</span><span class="o">.</span><span class="na">inc</span><span class="o">();</span>
+ <span class="k">return</span> <span class="n">value</span><span class="o">;</span>
+ <span class="o">}</span>
+<span class="o">}</span></code></pre></div>
<center><i><small>Excerpt from <a href="https://github.com/mbode/flink-prometheus-example/blob/master/src/main/java/com/github/mbode/flink_prometheus_example/FlinkMetricsExposingMapFunction.java">FlinkMetricsExposingMapFunction.java</a> demonstrating custom Flink metric.</small></i></center>
-## Configuring Prometheus with Flink
-
-To start monitoring Flink with Prometheus, the following steps are necessary:
-
-1. Make the `PrometheusReporter` jar available to the classpath of the Flink cluster (it comes with the Flink distribution):
+<h2 id="configuring-prometheus-with-flink">Configuring Prometheus with Flink</h2>
- cp /opt/flink/opt/flink-metrics-prometheus-1.7.2.jar /opt/flink/lib
+<p>To start monitoring Flink with Prometheus, the following steps are necessary:</p>
-2. [Configure the reporter]({{ site.DOCS_BASE_URL }}flink-docs-release-1.7/monitoring/metrics.html#reporter) in Flink's _flink-conf.yaml_. All job managers and task managers will expose the metrics on the configured port.
+<ol>
+ <li>
+ <p>Make the <code>PrometheusReporter</code> jar available to the classpath of the Flink cluster (it comes with the Flink distribution):</p>
- metrics.reporters: prom
- metrics.reporter.prom.class: org.apache.flink.metrics.prometheus.PrometheusReporter
- metrics.reporter.prom.port: 9999
+ <div class="highlight"><pre><code> cp /opt/flink/opt/flink-metrics-prometheus-1.7.2.jar /opt/flink/lib
+</code></pre></div>
+ </li>
+ <li>
+ <p><a href="https://ci.apache.org/projects/flink/flink-docs-release-1.7/monitoring/metrics.html#reporter">Configure the reporter</a> in Flink’s <em>flink-conf.yaml</em>. All job managers and task managers will expose the metrics on the configured port.</p>
-3. Prometheus needs to know where to scrape metrics. In a static scenario, you can simply [configure Prometheus](https://prometheus.io/docs/prometheus/latest/configuration/configuration/) in _prometheus.yml_ with the following:
+ <div class="highlight"><pre><code> metrics.reporters: prom
+ metrics.reporter.prom.class: org.apache.flink.metrics.prometheus.PrometheusReporter
+ metrics.reporter.prom.port: 9999
+</code></pre></div>
+ </li>
+ <li>
+ <p>Prometheus needs to know where to scrape metrics. In a static scenario, you can simply <a href="https://prometheus.io/docs/prometheus/latest/configuration/configuration/">configure Prometheus</a> in <em>prometheus.yml</em> with the following:</p>
- scrape_configs:
- - job_name: 'flink'
- static_configs:
- - targets: ['job-cluster:9999', 'taskmanager1:9999', 'taskmanager2:9999']
+ <div class="highlight"><pre><code> scrape_configs:
+ - job_name: 'flink'
+ static_configs:
+ - targets: ['job-cluster:9999', 'taskmanager1:9999', 'taskmanager2:9999']
+</code></pre></div>
- In more dynamic scenarios we recommend using Prometheus' service discovery support for different platforms such as Kubernetes, AWS EC2 and more.
+ <p>In more dynamic scenarios we recommend using Prometheus’ service discovery support for different platforms such as Kubernetes, AWS EC2 and more.</p>
+ </li>
+</ol>
-Both custom metrics are now available in Prometheus:
+<p>Both custom metrics are now available in Prometheus:</p>
<center>
-<img src="{{ site.baseurl }}/img/blog/2019-03-11-prometheus-monitoring/prometheus.png" width="600px" alt="Prometheus web UI with example metric"/>
-<br/>
+<img src="/img/blog/2019-03-11-prometheus-monitoring/prometheus.png" width="600px" alt="Prometheus web UI with example metric" />
+<br />
<i><small>Example metric in Prometheus web UI.</small></i>
</center>
-<br/>
+<p><br /></p>
-More technical metrics from the Flink cluster (like checkpoint sizes or duration, Kafka offsets or resource consumption) are also available. If you are interested, you can check out the HTTP endpoints exposing all Prometheus metrics for the job managers and the two task managers on [http://localhost:9249](http://localhost:9249/metrics), [http://localhost:9250](http://localhost:9250/metrics) and [http://localhost:9251](http://localhost:9251/metrics), respectively.
+<p>More technical metrics from the Flink cluster (like checkpoint sizes or duration, Kafka offsets or resource consumption) are also available. If you are interested, you can check out the HTTP endpoints exposing all Prometheus metrics for the job managers and the two task managers on <a href="http://localhost:9249/metrics">http://localhost:9249</a>, <a href="http://localhost:9250/metrics">http://localhost:9250</a> and <a href="ht [...]
-To test Prometheus' alerting feature, kill one of the Flink task managers via
+<p>To test Prometheus’ alerting feature, kill one of the Flink task managers via</p>
-```
-docker kill taskmanager1
-```
+<div class="highlight"><pre><code>docker kill taskmanager1
+</code></pre></div>
-Our Flink job can recover from this partial failure via the mechanism of [Checkpointing]({{ site.DOCS_BASE_URL }}flink-docs-release-1.7/dev/stream/state/checkpointing.html). Nevertheless, after roughly one minute (as configured in the alert rule) the following alert will fire:
+<p>Our Flink job can recover from this partial failure via the mechanism of <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.7/dev/stream/state/checkpointing.html">Checkpointing</a>. Nevertheless, after roughly one minute (as configured in the alert rule) the following alert will fire:</p>
<center>
-<img src="{{ site.baseurl }}/img/blog/2019-03-11-prometheus-monitoring/prometheusalerts.png" width="600px" alt="Prometheus web UI with example alert"/>
-<br/>
+<img src="/img/blog/2019-03-11-prometheus-monitoring/prometheusalerts.png" width="600px" alt="Prometheus web UI with example alert" />
+<br />
<i><small>Example alert in Prometheus web UI.</small></i>
</center>
-<br/>
+<p><br /></p>
-In real-world situations alerts like this one can be routed through a component called [Alertmanager](https://prometheus.io/docs/alerting/alertmanager/) and be grouped into notifications to systems like email, PagerDuty or Slack.
+<p>In real-world situations alerts like this one can be routed through a component called <a href="https://prometheus.io/docs/alerting/alertmanager/">Alertmanager</a> and be grouped into notifications to systems like email, PagerDuty or Slack.</p>
-Go ahead and play around with the setup, and check out the [Grafana](https://grafana.com/grafana) instance reachable at [http://localhost:3000](http://localhost:3000/) (credentials _admin:flink_) for visualizing Prometheus metrics. If there are any questions or problems, feel free to [create an issue](https://github.com/mbode/flink-prometheus-example/issues). Once finished, do not forget to tear down the setup via
+<p>Go ahead and play around with the setup, and check out the <a href="https://grafana.com/grafana">Grafana</a> instance reachable at <a href="http://localhost:3000/">http://localhost:3000</a> (credentials <em>admin:flink</em>) for visualizing Prometheus metrics. If there are any questions or problems, feel free to <a href="https://github.com/mbode/flink-prometheus-example/issues">create an issue</a>. Onc [...]
-```
-./gradlew composeDown
-```
-<br/>
+<div class="highlight"><pre><code>./gradlew composeDown
+</code></pre></div>
+<p><br /></p>
-## Conclusion
+<h2 id="conclusion">Conclusion</h2>
-Using Prometheus together with Flink provides an easy way for effective monitoring and alerting of your Flink jobs. Both projects have exciting and vibrant communities behind them with new developments and additions scheduled for upcoming releases. We encourage you to try the two technologies together as it has immensely improved our insights into Flink jobs running in production.
+<p>Using Prometheus together with Flink provides an easy way for effective monitoring and alerting of your Flink jobs. Both projects have exciting and vibrant communities behind them with new developments and additions scheduled for upcoming releases. We encourage you to try the two technologies together as it has immensely improved our insights into Flink jobs running in production.</p>
</description>
-<pubDate>Mon, 11 Mar 2019 12:00:00 +0000</pubDate>
+<pubDate>Mon, 11 Mar 2019 13:00:00 +0100</pubDate>
<link>https://flink.apache.org/features/2019/03/11/prometheus-monitoring.html</link>
<guid isPermaLink="true">/features/2019/03/11/prometheus-monitoring.html</guid>
</item>
<item>
<title>What to expect from Flink Forward San Francisco 2019</title>
-<description>The third annual Flink Forward San Francisco is just a few weeks away! As always, Flink Forward will be the right place to meet and mingle with experienced Flink users, contributors, and committers. Attendees will hear and chat about the latest developments around Flink and learn from technical deep-dive sessions and exciting use cases that were put into production with Flink. The event will take place on April 1-2, 2019 at Hotel Nikko in San Francisco. The [program committe [...]
+<description><p>The third annual Flink Forward San Francisco is just a few weeks away! As always, Flink Forward will be the right place to meet and mingle with experienced Flink users, contributors, and committers. Attendees will hear and chat about the latest developments around Flink and learn from technical deep-dive sessions and exciting use cases that were put into production with Flink. The event will take place on April 1-2, 2019 at Hotel Nikko in San Francisco. The <a hr [...]
-Some highlights of the program are:
+<p>Some highlights of the program are:</p>
-* [Realtime Store Visit Predictions at Scale](https://sf-2019.flink-forward.org/conference-program#realtime-store-visit-predictions-at-scale): Luca Giovagnoli from *Yelp* will talk about a "multidisciplinary" Flink application that combines geospatial clustering algorithms, Machine Learning models, and cutting-edge stream-processing technology.
-
-* [Real-time Processing with Flink for Machine Learning at Netflix](https://sf-2019.flink-forward.org/conference-program#real-time-processing-with-flink-for-machine-learning-at-netflix): Elliot Chow will discuss the practical aspects of using Apache Flink to power Machine Learning algorithms for video recommendations, search results ranking, and selection of artwork images at *Netflix*.
-
-* [Building production Flink jobs with Airstream at Airbnb](https://sf-2019.flink-forward.org/conference-program#building-production-flink-jobs-with-airstream-at-airbnb): Pala Muthiah and Hao Wang will reveal how *Airbnb* builds real time data pipelines with Airstream, Airbnb's computation framework that is powered by Flink SQL.
-
-* [When Table meets AI: Build Flink AI Ecosystem on Table API](https://sf-2019.flink-forward.org/conference-program#when-table-meets-ai--build-flink-ai-ecosystem-on-table-api): Shaoxuan Wang from *Alibaba* will discuss how they are building a solid AI ecosystem for unified batch/streaming Machine Learning data pipelines on top of Flink's Table API.
-
-* [Adventures in Scaling from Zero to 5 Billion Data Points per Day](https://sf-2019.flink-forward.org/conference-program#adventures-in-scaling-from-zero-to-5-billion-data-points-per-day): Dave Torok will take us through *Comcast's* journey in scaling the company's operationalized Machine Learning framework from the very early days in production to processing more than 5 billion data points per day.
-
-If you're new to Apache Flink or want to deepen your knowledge around the framework, Flink Forward features again a full day of training.
-
-You can choose from 3 training tracks:
+<ul>
+ <li>
+ <p><a href="https://sf-2019.flink-forward.org/conference-program#realtime-store-visit-predictions-at-scale">Realtime Store Visit Predictions at Scale</a>: Luca Giovagnoli from <em>Yelp</em> will talk about a “multidisciplinary” Flink application that combines geospatial clustering algorithms, Machine Learning models, and cutting-edge stream-processing technology.</p>
+ </li>
+ <li>
+ <p><a href="https://sf-2019.flink-forward.org/conference-program#real-time-processing-with-flink-for-machine-learning-at-netflix">Real-time Processing with Flink for Machine Learning at Netflix</a>: Elliot Chow will discuss the practical aspects of using Apache Flink to power Machine Learning algorithms for video recommendations, search results ranking, and selection of artwork images at <em>Netflix</em>.</p>
+ </li>
+ <li>
+ <p><a href="https://sf-2019.flink-forward.org/conference-program#building-production-flink-jobs-with-airstream-at-airbnb">Building production Flink jobs with Airstream at Airbnb</a>: Pala Muthiah and Hao Wang will reveal how <em>Airbnb</em> builds real time data pipelines with Airstream, Airbnb’s computation framework that is powered by Flink SQL.</p>
+ </li>
+ <li>
+ <p><a href="https://sf-2019.flink-forward.org/conference-program#when-table-meets-ai--build-flink-ai-ecosystem-on-table-api">When Table meets AI: Build Flink AI Ecosystem on Table API</a>: Shaoxuan Wang from <em>Alibaba</em> will discuss how they are building a solid AI ecosystem for unified batch/streaming Machine Learning data pipelines on top of Flink’s Table API.</p>
+ </li>
+ <li>
+ <p><a href="https://sf-2019.flink-forward.org/conference-program#adventures-in-scaling-from-zero-to-5-billion-data-points-per-day">Adventures in Scaling from Zero to 5 Billion Data Points per Day</a>: Dave Torok will take us through <em>Comcast’s</em> journey in scaling the company’s operationalized Machine Learning framework from the very early days in production to processing more than 5 billion data points per day.</p>
+ </li>
+</ul>
-* [Introduction to Streaming with Apache Flink](https://sf-2019.flink-forward.org/training-program#introduction-to-streaming-with-apache-flink): A hands-on, in-depth introduction to stream processing and Apache Flink, this course emphasizes those features of Flink that make it easy to build and manage accurate, fault tolerant applications on streams.
+<p>If you’re new to Apache Flink or want to deepen your knowledge around the framework, Flink Forward features again a full day of training.</p>
-* [Analyzing Streaming Data with Flink SQL](https://sf-2019.flink-forward.org/training-program#analyzing-streaming-data-with-flink-sql): In this hands-on training, you will learn what it means to run SQL queries on data streams and how to fully leverage the potential of SQL on Flink. We'll also cover some of the more recent features such as time-versioned joins and the MATCH RECOGNIZE clause.
+<p>You can choose from 3 training tracks:</p>
-* [Troubleshooting and Operating Flink at large scale](https://sf-2019.flink-forward.org/training-program#apache-flink-troubleshooting---operations): In this training, we will focus on everything you need to run Apache Flink applications reliably and efficiently in production including topics like capacity planning, monitoring, troubleshooting and tuning Apache Flink.
+<ul>
+ <li>
+ <p><a href="https://sf-2019.flink-forward.org/training-program#introduction-to-streaming-with-apache-flink">Introduction to Streaming with Apache Flink</a>: A hands-on, in-depth introduction to stream processing and Apache Flink, this course emphasizes those features of Flink that make it easy to build and manage accurate, fault tolerant applications on streams.</p>
+ </li>
+ <li>
+ <p><a href="https://sf-2019.flink-forward.org/training-program#analyzing-streaming-data-with-flink-sql">Analyzing Streaming Data with Flink SQL</a>: In this hands-on training, you will learn what it means to run SQL queries on data streams and how to fully leverage the potential of SQL on Flink. We’ll also cover some of the more recent features such as time-versioned joins and the MATCH RECOGNIZE clause.</p>
+ </li>
+ <li>
+ <p><a href="https://sf-2019.flink-forward.org/training-program#apache-flink-troubleshooting---operations">Troubleshooting and Operating Flink at large scale</a>: In this training, we will focus on everything you need to run Apache Flink applications reliably and efficiently in production including topics like capacity planning, monitoring, troubleshooting and tuning Apache Flink.</p>
+ </li>
+</ul>
-If you haven't done so yet, check out the [full schedule](http://sf-2019.flink-forward.org/conference-program) and [register](https://sf-2019.flink-forward.org/register) your attendance. <br>
-I'm looking forward to meet you at Flink Forward San Francisco.
+<p>If you haven’t done so yet, check out the <a href="http://sf-2019.flink-forward.org/conference-program">full schedule</a> and <a href="https://sf-2019.flink-forward.org/register">register</a> your attendance. <br />
+I’m looking forward to meet you at Flink Forward San Francisco.</p>
-*Fabian*</description>
-<pubDate>Wed, 06 Mar 2019 11:00:00 +0000</pubDate>
+<p><em>Fabian</em></p>
+</description>
+<pubDate>Wed, 06 Mar 2019 12:00:00 +0100</pubDate>
<link>https://flink.apache.org/news/2019/03/06/ffsf-preview.html</link>
<guid isPermaLink="true">/news/2019/03/06/ffsf-preview.html</guid>
</item>
@@ -5026,163 +5226,210 @@ I'm looking forward to meet you at Flink Forward San Francisco.
td { vertical-align: top }
</style>
-This blog post provides an introduction to Apache Flink’s built-in monitoring
+<p>This blog post provides an introduction to Apache Flink’s built-in monitoring
and metrics system, that allows developers to effectively monitor their Flink
jobs. Oftentimes, the task of picking the relevant metrics to monitor a
Flink application can be overwhelming for a DevOps team that is just starting
with stream processing and Apache Flink. Having worked with many organizations
that deploy Flink at scale, I would like to share my experience and some best
-practice with the community.
+practice with the community.</p>
-With business-critical applications running on Apache Flink, performance monitoring
+<p>With business-critical applications running on Apache Flink, performance monitoring
becomes an increasingly important part of a successful production deployment. It
ensures that any degradation or downtime is immediately identified and resolved
-as quickly as possible.
+as quickly as possible.</p>
-Monitoring goes hand-in-hand with observability, which is a prerequisite for
+<p>Monitoring goes hand-in-hand with observability, which is a prerequisite for
troubleshooting and performance tuning. Nowadays, with the complexity of modern
enterprise applications and the speed of delivery increasing, an engineering
team must understand and have a complete overview of its applications’ status at
-any given point in time.
+any given point in time.</p>
-## Flink’s Metrics System
+<h2 id="flinks-metrics-system">Flink’s Metrics System</h2>
-The foundation for monitoring Flink jobs is its [metrics
-system](<{{ site.DOCS_BASE_URL }}flink-docs-release-1.7/monitoring/metrics.html>)
-which consists of two components; `Metrics` and `MetricsReporters`.
+<p>The foundation for monitoring Flink jobs is its <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.7/monitoring/metrics.html">metrics
+system</a>
+which consists of two components; <code>Metrics</code> and <code>MetricsReporters</code>.</p>
-### Metrics
+<h3 id="metrics">Metrics</h3>
-Flink comes with a comprehensive set of built-in metrics such as:
+<p>Flink comes with a comprehensive set of built-in metrics such as:</p>
-* Used JVM Heap / NonHeap / Direct Memory (per Task-/JobManager)
-* Number of Job Restarts (per Job)
-* Number of Records Per Second (per Operator)
-* ...
+<ul>
+ <li>Used JVM Heap / NonHeap / Direct Memory (per Task-/JobManager)</li>
+ <li>Number of Job Restarts (per Job)</li>
+ <li>Number of Records Per Second (per Operator)</li>
+ <li>…</li>
+</ul>
-These metrics have different scopes and measure more general (e.g. JVM or
-operating system) as well as Flink-specific aspects.
+<p>These metrics have different scopes and measure more general (e.g. JVM or
+operating system) as well as Flink-specific aspects.</p>
-As a user, you can and should add application-specific metrics to your
+<p>As a user, you can and should add application-specific metrics to your
functions. Typically these include counters for the number of invalid records or
the number of records temporarily buffered in managed state. Besides counters,
Flink offers additional metrics types like gauges and histograms. For
instructions on how to register your own metrics with Flink’s metrics system
-please check out [Flink’s
-documentation](<{{ site.DOCS_BASE_URL }}flink-docs-release-1.7/monitoring/metrics.html#registering-metrics>).
+please check out <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.7/monitoring/metrics.html#registering-metrics">Flink’s
+documentation</a>.
In this blog post, we will focus on how to get the most out of Flink’s built-in
-metrics.
+metrics.</p>
-### MetricsReporters
+<h3 id="metricsreporters">MetricsReporters</h3>
-All metrics can be queried via Flink’s REST API. However, users can configure
+<p>All metrics can be queried via Flink’s REST API. However, users can configure
MetricsReporters to send the metrics to external systems. Apache Flink provides
reporters to the most common monitoring tools out-of-the-box including JMX,
Prometheus, Datadog, Graphite and InfluxDB. For information about how to
-configure a reporter check out Flink’s [MetricsReporter
-documentation](<{{ site.DOCS_BASE_URL }}flink-docs-release-1.7/monitoring/metrics.html#reporter>).
+configure a reporter check out Flink’s <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.7/monitoring/metrics.html#reporter">MetricsReporter
+documentation</a>.</p>
-In the remaining part of this blog post, we will go over some of the most
-important metrics to monitor your Apache Flink application.
+<p>In the remaining part of this blog post, we will go over some of the most
+important metrics to monitor your Apache Flink application.</p>
-## Monitoring General Health
+<h2 id="monitoring-general-health">Monitoring General Health</h2>
-The first thing you want to monitor is whether your job is actually in a *RUNNING*
+<p>The first thing you want to monitor is whether your job is actually in a <em>RUNNING</em>
state. In addition, it pays off to monitor the number of restarts and the time
-since the last restart.
+since the last restart.</p>
-Generally speaking, successful checkpointing is a strong indicator of the
+<p>Generally speaking, successful checkpointing is a strong indicator of the
general health of your application. For each checkpoint, checkpoint barriers
need to flow through the whole topology of your Flink job and events and
barriers cannot overtake each other. Therefore, a successful checkpoint shows
-that no channel is fully congested.
+that no channel is fully congested.</p>
-**Key Metrics**
+<p><strong>Key Metrics</strong></p>
-Metric | Scope | Description |
------- | ----- | ----------- |
-`uptime` | job | The time that the job has been running without interruption. |
-`fullRestarts` | job | The total number of full restarts since this job was submitted. |
-`numberOfCompletedCheckpoints` | job | The number of successfully completed checkpoints. |
-`numberOfFailedCheckpoints` | job | The number of failed checkpoints. |
+<table>
+ <thead>
+ <tr>
+ <th>Metric</th>
+ <th>Scope</th>
+ <th>Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><code>uptime</code></td>
+ <td>job</td>
+ <td>The time that the job has been running without interruption.</td>
+ </tr>
+ <tr>
+ <td><code>fullRestarts</code></td>
+ <td>job</td>
+ <td>The total number of full restarts since this job was submitted.</td>
+ </tr>
+ <tr>
+ <td><code>numberOfCompletedCheckpoints</code></td>
+ <td>job</td>
+ <td>The number of successfully completed checkpoints.</td>
+ </tr>
+ <tr>
+ <td><code>numberOfFailedCheckpoints</code></td>
+ <td>job</td>
+ <td>The number of failed checkpoints.</td>
+ </tr>
+ </tbody>
+</table>
-<br/>
+<p><br /></p>
-**Example Dashboard Panels**
+<p><strong>Example Dashboard Panels</strong></p>
<center>
-<img src="{{ site.baseurl }}/img/blog/2019-02-21-monitoring-best-practices/fig-1.png" width="800px" alt="Uptime (35 minutes), Restarting Time (3 milliseconds) and Number of Full Restarts (7)"/>
-<br/>
+<img src="/img/blog/2019-02-21-monitoring-best-practices/fig-1.png" width="800px" alt="Uptime (35 minutes), Restarting Time (3 milliseconds) and Number of Full Restarts (7)" />
+<br />
<i><small>Uptime (35 minutes), Restarting Time (3 milliseconds) and Number of Full Restarts (7)</small></i>
</center>
-<br/>
+<p><br /></p>
<center>
-<img src="{{ site.baseurl }}/img/blog/2019-02-21-monitoring-best-practices/fig-2.png" width="800px" alt="Completed Checkpoints (18336), Failed (14)"/>
-<br/>
+<img src="/img/blog/2019-02-21-monitoring-best-practices/fig-2.png" width="800px" alt="Completed Checkpoints (18336), Failed (14)" />
+<br />
<i><small>Completed Checkpoints (18336), Failed (14)</small></i>
</center>
-<br/>
+<p><br /></p>
-**Possible Alerts**
-
-* `ΔfullRestarts` &gt; `threshold`
-* `ΔnumberOfFailedCheckpoints` &gt; `threshold`
+<p><strong>Possible Alerts</strong></p>
+<ul>
+ <li><code>ΔfullRestarts</code> &gt; <code>threshold</code></li>
+ <li><code>ΔnumberOfFailedCheckpoints</code> &gt; <code>threshold</code></li>
+</ul>
-## Monitoring Progress & Throughput
+<h2 id="monitoring-progress--throughput">Monitoring Progress &amp; Throughput</h2>
-Knowing that your application is RUNNING and checkpointing is working fine is good,
+<p>Knowing that your application is RUNNING and checkpointing is working fine is good,
but it does not tell you whether the application is actually making progress and
-keeping up with the upstream systems.
+keeping up with the upstream systems.</p>
-### Throughput
+<h3 id="throughput">Throughput</h3>
-Flink provides multiple metrics to measure the throughput of our application.
-For each operator or task (remember: a task can contain multiple [chained
-tasks](<{{ site.DOCS_BASE_URL }}flink-docs-release-1.7/dev/stream/operators/#task-chaining-and-resource-groups>)
+<p>Flink provides multiple metrics to measure the throughput of our application.
+For each operator or task (remember: a task can contain multiple <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.7/dev/stream/operators/#task-chaining-and-resource-groups">chained
+tasks</a>
Flink counts the number of records and bytes going in and out. Out of those
metrics, the rate of outgoing records per operator is often the most intuitive
-and easiest to reason about.
+and easiest to reason about.</p>
-**Key Metrics**
+<p><strong>Key Metrics</strong></p>
-Metric | Scope | Description |
------- | ----- | ----------- |
-`numRecordsOutPerSecond` | task | The number of records this operator/task sends per second. |
-`numRecordsOutPerSecond` | operator | The number of records this operator sends per second. |
+<table>
+ <thead>
+ <tr>
+ <th>Metric</th>
+ <th>Scope</th>
+ <th>Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><code>numRecordsOutPerSecond</code></td>
+ <td>task</td>
+ <td>The number of records this operator/task sends per second.</td>
+ </tr>
+ <tr>
+ <td><code>numRecordsOutPerSecond</code></td>
+ <td>operator</td>
+ <td>The number of records this operator sends per second.</td>
+ </tr>
+ </tbody>
+</table>
-<br/>
+<p><br /></p>
-**Example Dashboard Panels**
+<p><strong>Example Dashboard Panels</strong></p>
<center>
-<img src="{{ site.baseurl }}/img/blog/2019-02-21-monitoring-best-practices/fig-3.png" width="800px" alt="Mean Records Out per Second per Operator"/>
-<br/>
+<img src="/img/blog/2019-02-21-monitoring-best-practices/fig-3.png" width="800px" alt="Mean Records Out per Second per Operator" />
+<br />
<i><small>Mean Records Out per Second per Operator</small></i>
</center>
-<br/>
+<p><br /></p>
-**Possible Alerts**
+<p><strong>Possible Alerts</strong></p>
-* `recordsOutPerSecond` = `0` (for a non-Sink operator)
+<ul>
+ <li><code>recordsOutPerSecond</code> = <code>0</code> (for a non-Sink operator)</li>
+</ul>
-_Note_: Source operators always have zero incoming records. Sink operators
+<p><em>Note</em>: Source operators always have zero incoming records. Sink operators
always have zero outgoing records because the metrics only count
-Flink-internal communication. There is a [JIRA
-ticket](<https://issues.apache.org/jira/browse/FLINK-7286>) to change this
-behavior.
+Flink-internal communication. There is a <a href="https://issues.apache.org/jira/browse/FLINK-7286">JIRA
+ticket</a> to change this
+behavior.</p>
-### Progress
+<h3 id="progress">Progress</h3>
-For applications, that use event time semantics, it is important that watermarks
-progress over time. A watermark of time _t_ tells the framework, that it
-should not anymore expect to receive events with a timestamp earlier than _t_,
-and in turn, to trigger all operations that were scheduled for a timestamp &lt; _t_.
-For example, an event time window that ends at _t_ = 30 will be closed and
-evaluated once the watermark passes 30.
+<p>For applications, that use event time semantics, it is important that watermarks
+progress over time. A watermark of time <em>t</em> tells the framework, that it
+should not anymore expect to receive events with a timestamp earlier than <em>t</em>,
+and in turn, to trigger all operations that were scheduled for a timestamp &lt; <em>t</em>.
+For example, an event time window that ends at <em>t</em> = 30 will be closed and
+evaluated once the watermark passes 30.</p>
-As a consequence, you should monitor the watermark at event time-sensitive
+<p>As a consequence, you should monitor the watermark at event time-sensitive
operators in your application, such as process functions and windows. If the
difference between the current processing time and the watermark, known as
even-time skew, is unusually high, then it typically implies one of two issues.
@@ -5191,195 +5438,294 @@ during catch-up after a downtime or when your job is simply not able to keep up
and events are queuing up. Second, it could mean a single upstream sub-task has
not sent a watermark for a long time (for example because it did not receive any
events to base the watermark on), which also prevents the watermark in
-downstream operators to progress. This [JIRA
-ticket](<https://issues.apache.org/jira/browse/FLINK-5017>) provides further
-information and a work around for the latter.
+downstream operators to progress. This <a href="https://issues.apache.org/jira/browse/FLINK-5017">JIRA
+ticket</a> provides further
+information and a work around for the latter.</p>
-**Key Metrics**
+<p><strong>Key Metrics</strong></p>
-Metric | Scope | Description |
------- | ----- | ----------- |
-`currentOutputWatermark` | operator | The last watermark this operator has emitted. |
+<table>
+ <thead>
+ <tr>
+ <th>Metric</th>
+ <th>Scope</th>
+ <th>Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><code>currentOutputWatermark</code></td>
+ <td>operator</td>
+ <td>The last watermark this operator has emitted.</td>
+ </tr>
+ </tbody>
+</table>
-<br/>
+<p><br /></p>
-**Example Dashboard Panels**
+<p><strong>Example Dashboard Panels</strong></p>
<center>
-<img src="{{ site.baseurl }}/img/blog/2019-02-21-monitoring-best-practices/fig-4.png" width="800px" alt="Event Time Lag per Subtask of a single operator in the topology. In this case, the watermark is lagging a few seconds behind for each subtask."/>
-<br/>
+<img src="/img/blog/2019-02-21-monitoring-best-practices/fig-4.png" width="800px" alt="Event Time Lag per Subtask of a single operator in the topology. In this case, the watermark is lagging a few seconds behind for each subtask." />
+<br />
<i><small>Event Time Lag per Subtask of a single operator in the topology. In this case, the watermark is lagging a few seconds behind for each subtask.</small></i>
</center>
-<br/>
+<p><br /></p>
-**Possible Alerts**
+<p><strong>Possible Alerts</strong></p>
-* `currentProcessingTime - currentOutputWatermark` &gt; `threshold`
+<ul>
+ <li><code>currentProcessingTime - currentOutputWatermark</code> &gt; <code>threshold</code></li>
+</ul>
-### "Keeping Up"
+<h3 id="keeping-up">“Keeping Up”</h3>
-When consuming from a message queue, there is often a direct way to monitor if
+<p>When consuming from a message queue, there is often a direct way to monitor if
your application is keeping up. By using connector-specific metrics you can
monitor how far behind the head of the message queue your current consumer group
-is. Flink forwards the underlying metrics from most sources.
+is. Flink forwards the underlying metrics from most sources.</p>
-**Key Metrics**
+<p><strong>Key Metrics</strong></p>
-Metric | Scope | Description |
------- | ----- | ----------- |
-`records-lag-max` | user | applies to `FlinkKafkaConsumer`. The maximum lag in terms of the number of records for any partition in this window. An increasing value over time is your best indication that the consumer group is not keeping up with the producers. |
-`millisBehindLatest` | user | applies to `FlinkKinesisConsumer`. The number of milliseconds a consumer is behind the head of the stream. For any consumer and Kinesis shard, this indicates how far it is behind the current time. |
+<table>
+ <thead>
+ <tr>
+ <th>Metric</th>
+ <th>Scope</th>
+ <th>Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><code>records-lag-max</code></td>
+ <td>user</td>
+ <td>applies to <code>FlinkKafkaConsumer</code>. The maximum lag in terms of the number of records for any partition in this window. An increasing value over time is your best indication that the consumer group is not keeping up with the producers.</td>
+ </tr>
+ <tr>
+ <td><code>millisBehindLatest</code></td>
+ <td>user</td>
+ <td>applies to <code>FlinkKinesisConsumer</code>. The number of milliseconds a consumer is behind the head of the stream. For any consumer and Kinesis shard, this indicates how far it is behind the current time.</td>
+ </tr>
+ </tbody>
+</table>
-<br/>
+<p><br /></p>
-**Possible Alerts**
+<p><strong>Possible Alerts</strong></p>
-* `records-lag-max` &gt; `threshold`
-* `millisBehindLatest` &gt; `threshold`
+<ul>
+ <li><code>records-lag-max</code> &gt; <code>threshold</code></li>
+ <li><code>millisBehindLatest</code> &gt; <code>threshold</code></li>
+</ul>
-## Monitoring Latency
+<h2 id="monitoring-latency">Monitoring Latency</h2>
-Generally speaking, latency is the delay between the creation of an event and
+<p>Generally speaking, latency is the delay between the creation of an event and
the time at which results based on this event become visible. Once the event is
created it is usually stored in a persistent message queue, before it is
processed by Apache Flink, which then writes the results to a database or calls
a downstream system. In such a pipeline, latency can be introduced at each stage
-and for various reasons including the following:
-
-1. It might take a varying amount of time until events are persisted in the
-message queue.
-2. During periods of high load or during recovery, events might spend some time
-in the message queue until they are processed by Flink (see previous section).
-3. Some operators in a streaming topology need to buffer events for some time
-(e.g. in a time window) for functional reasons.
-4. Each computation in your Flink topology (framework or user code), as well as
-each network shuffle, takes time and adds to latency.
-5. If the application emits through a transactional sink, the sink will only
+and for various reasons including the following:</p>
+
+<ol>
+ <li>It might take a varying amount of time until events are persisted in the
+message queue.</li>
+ <li>During periods of high load or during recovery, events might spend some time
+in the message queue until they are processed by Flink (see previous section).</li>
+ <li>Some operators in a streaming topology need to buffer events for some time
+(e.g. in a time window) for functional reasons.</li>
+ <li>Each computation in your Flink topology (framework or user code), as well as
+each network shuffle, takes time and adds to latency.</li>
+ <li>If the application emits through a transactional sink, the sink will only
commit and publish transactions upon successful checkpoints of Flink, adding
-latency usually up to the checkpointing interval for each record.
+latency usually up to the checkpointing interval for each record.</li>
+</ol>
-In practice, it has proven invaluable to add timestamps to your events at
+<p>In practice, it has proven invaluable to add timestamps to your events at
multiple stages (at least at creation, persistence, ingestion by Flink,
publication by Flink, possibly sampling those to save bandwidth). The
differences between these timestamps can be exposed as a user-defined metric in
-your Flink topology to derive the latency distribution of each stage.
+your Flink topology to derive the latency distribution of each stage.</p>
-In the rest of this section, we will only consider latency, which is introduced
+<p>In the rest of this section, we will only consider latency, which is introduced
inside the Flink topology and cannot be attributed to transactional sinks or
-events being buffered for functional reasons (4.).
+events being buffered for functional reasons (4.).</p>
-To this end, Flink comes with a feature called [Latency
-Tracking](<{{ site.DOCS_BASE_URL }}flink-docs-release-1.7/monitoring/metrics.html#latency-tracking>).
+<p>To this end, Flink comes with a feature called <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.7/monitoring/metrics.html#latency-tracking">Latency
+Tracking</a>.
When enabled, Flink will insert so-called latency markers periodically at all
sources. For each sub-task, a latency distribution from each source to this
operator will be reported. The granularity of these histograms can be further
-controlled by setting _metrics.latency.granularity_ as desired.
+controlled by setting <em>metrics.latency.granularity</em> as desired.</p>
-Due to the potentially high number of histograms (in particular for
-_metrics.latency.granularity: subtask_), enabling latency tracking can
+<p>Due to the potentially high number of histograms (in particular for
+<em>metrics.latency.granularity: subtask</em>), enabling latency tracking can
significantly impact the performance of the cluster. It is recommended to only
-enable it to locate sources of latency during debugging.
+enable it to locate sources of latency during debugging.</p>
-**Metrics**
+<p><strong>Metrics</strong></p>
-Metric | Scope | Description |
------- | ----- | ----------- |
-`latency` | operator | The latency from the source operator to this operator. |
-`restartingTime` | job | The time it took to restart the job, or how long the current restart has been in progress. |
+<table>
+ <thead>
+ <tr>
+ <th>Metric</th>
+ <th>Scope</th>
+ <th>Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><code>latency</code></td>
+ <td>operator</td>
+ <td>The latency from the source operator to this operator.</td>
+ </tr>
+ <tr>
+ <td><code>restartingTime</code></td>
+ <td>job</td>
+ <td>The time it took to restart the job, or how long the current restart has been in progress.</td>
+ </tr>
+ </tbody>
+</table>
-<br/>
+<p><br /></p>
-**Example Dashboard Panel**
+<p><strong>Example Dashboard Panel</strong></p>
<center>
-<img src="{{ site.baseurl }}/img/blog/2019-02-21-monitoring-best-practices/fig-5.png" width="800px" alt="Latency distribution between a source and a single sink subtask."/>
-<br/>
+<img src="/img/blog/2019-02-21-monitoring-best-practices/fig-5.png" width="800px" alt="Latency distribution between a source and a single sink subtask." />
+<br />
<i><small>Latency distribution between a source and a single sink subtask.</small></i>
</center>
-<br/>
+<p><br /></p>
-## JVM Metrics
+<h2 id="jvm-metrics">JVM Metrics</h2>
-So far we have only looked at Flink-specific metrics. As long as latency &
+<p>So far we have only looked at Flink-specific metrics. As long as latency &amp;
throughput of your application are in line with your expectations and it is
checkpointing consistently, this is probably everything you need. On the other
hand, if you job’s performance is starting to degrade among the firstmetrics you
-want to look at are memory consumption and CPU load of your Task- & JobManager
-JVMs.
+want to look at are memory consumption and CPU load of your Task- &amp; JobManager
+JVMs.</p>
-### Memory
+<h3 id="memory">Memory</h3>
-Flink reports the usage of Heap, NonHeap, Direct & Mapped memory for JobManagers
-and TaskManagers.
+<p>Flink reports the usage of Heap, NonHeap, Direct &amp; Mapped memory for JobManagers
+and TaskManagers.</p>
-* Heap memory - as with most JVM applications - is the most volatile and important
+<ul>
+ <li>
+ <p>Heap memory - as with most JVM applications - is the most volatile and important
metric to watch. This is especially true when using Flink’s filesystem
statebackend as it keeps all state objects on the JVM Heap. If the size of
long-living objects on the Heap increases significantly, this can usually be
attributed to the size of your application state (check the
-[checkpointing metrics](<{{ site.DOCS_BASE_URL }}flink-docs-release-1.7/monitoring/metrics.html#checkpointing>)
+<a href="https://ci.apache.org/projects/flink/flink-docs-release-1.7/monitoring/metrics.html#checkpointing">checkpointing metrics</a>
for an estimated size of the on-heap state). The possible reasons for growing
state are very application-specific. Typically, an increasing number of keys, a
large event-time skew between different input streams or simply missing state
-cleanup may cause growing state.
-
-* NonHeap memory is dominated by the metaspace, the size of which is unlimited by default
+cleanup may cause growing state.</p>
+ </li>
+ <li>
+ <p>NonHeap memory is dominated by the metaspace, the size of which is unlimited by default
and holds class metadata as well as static content. There is a
-[JIRA Ticket](<https://issues.apache.org/jira/browse/FLINK-10317>) to limit the size
-to 250 megabyte by default.
-
-* The biggest driver of Direct memory is by far the
+<a href="https://issues.apache.org/jira/browse/FLINK-10317">JIRA Ticket</a> to limit the size
+to 250 megabyte by default.</p>
+ </li>
+ <li>
+ <p>The biggest driver of Direct memory is by far the
number of Flink’s network buffers, which can be
-[configured](<{{ site.DOCS_BASE_URL }}flink-docs-release-1.7/ops/config.html#configuring-the-network-buffers>).
-
-* Mapped memory is usually close to zero as Flink does not use memory-mapped files.
+<a href="https://ci.apache.org/projects/flink/flink-docs-release-1.7/ops/config.html#configuring-the-network-buffers">configured</a>.</p>
+ </li>
+ <li>
+ <p>Mapped memory is usually close to zero as Flink does not use memory-mapped files.</p>
+ </li>
+</ul>
-In a containerized environment you should additionally monitor the overall
+<p>In a containerized environment you should additionally monitor the overall
memory consumption of the Job- and TaskManager containers to ensure they don’t
exceed their resource limits. This is particularly important, when using the
RocksDB statebackend, since RocksDB allocates a considerable amount of
memory off heap. To understand how much memory RocksDB might use, you can
-checkout [this blog
-post](<https://www.da-platform.com/blog/manage-rocksdb-memory-size-apache-flink>)
-by Stefan Richter.
+checkout <a href="https://www.da-platform.com/blog/manage-rocksdb-memory-size-apache-flink">this blog
+post</a>
+by Stefan Richter.</p>
-**Key Metrics**
+<p><strong>Key Metrics</strong></p>
-Metric | Scope | Description |
------- | ----- | ----------- |
-`Status.JVM.Memory.NonHeap.Committed` | job-/taskmanager | The amount of non-heap memory guaranteed to be available to the JVM (in bytes). |
-`Status.JVM.Memory.Heap.Used` | job-/taskmanager | The amount of heap memory currently used (in bytes). |
-`Status.JVM.Memory.Heap.Committed` | job-/taskmanager | The amount of heap memory guaranteed to be available to the JVM (in bytes). |
-`Status.JVM.Memory.Direct.MemoryUsed` | job-/taskmanager | The amount of memory used by the JVM for the direct buffer pool (in bytes). |
-`Status.JVM.Memory.Mapped.MemoryUsed` | job-/taskmanager | The amount of memory used by the JVM for the mapped buffer pool (in bytes). |
-`Status.JVM.GarbageCollector.G1 Young Generation.Time` | job-/taskmanager | The total time spent performing G1 Young Generation garbage collection. |
-`Status.JVM.GarbageCollector.G1 Old Generation.Time` | job-/taskmanager | The total time spent performing G1 Old Generation garbage collection. |
+<table>
+ <thead>
+ <tr>
+ <th>Metric</th>
+ <th>Scope</th>
+ <th>Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><code>Status.JVM.Memory.NonHeap.Committed</code></td>
+ <td>job-/taskmanager</td>
+ <td>The amount of non-heap memory guaranteed to be available to the JVM (in bytes).</td>
+ </tr>
+ <tr>
+ <td><code>Status.JVM.Memory.Heap.Used</code></td>
+ <td>job-/taskmanager</td>
+ <td>The amount of heap memory currently used (in bytes).</td>
+ </tr>
+ <tr>
+ <td><code>Status.JVM.Memory.Heap.Committed</code></td>
+ <td>job-/taskmanager</td>
+ <td>The amount of heap memory guaranteed to be available to the JVM (in bytes).</td>
+ </tr>
+ <tr>
+ <td><code>Status.JVM.Memory.Direct.MemoryUsed</code></td>
+ <td>job-/taskmanager</td>
+ <td>The amount of memory used by the JVM for the direct buffer pool (in bytes).</td>
+ </tr>
+ <tr>
+ <td><code>Status.JVM.Memory.Mapped.MemoryUsed</code></td>
+ <td>job-/taskmanager</td>
+ <td>The amount of memory used by the JVM for the mapped buffer pool (in bytes).</td>
+ </tr>
+ <tr>
+ <td><code>Status.JVM.GarbageCollector.G1 Young Generation.Time</code></td>
+ <td>job-/taskmanager</td>
+ <td>The total time spent performing G1 Young Generation garbage collection.</td>
+ </tr>
+ <tr>
+ <td><code>Status.JVM.GarbageCollector.G1 Old Generation.Time</code></td>
+ <td>job-/taskmanager</td>
+ <td>The total time spent performing G1 Old Generation garbage collection.</td>
+ </tr>
+ </tbody>
+</table>
-<br/>
+<p><br /></p>
-**Example Dashboard Panel**
+<p><strong>Example Dashboard Panel</strong></p>
<center>
-<img src="{{ site.baseurl }}/img/blog/2019-02-21-monitoring-best-practices/fig-6.png" width="800px" alt="TaskManager memory consumption and garbage collection times."/>
-<br/>
+<img src="/img/blog/2019-02-21-monitoring-best-practices/fig-6.png" width="800px" alt="TaskManager memory consumption and garbage collection times." />
+<br />
<i><small>TaskManager memory consumption and garbage collection times.</small></i>
</center>
-<br/>
+<p><br /></p>
<center>
-<img src="{{ site.baseurl }}/img/blog/2019-02-21-monitoring-best-practices/fig-7.png" width="800px" alt="JobManager memory consumption and garbage collection times."/>
-<br/>
+<img src="/img/blog/2019-02-21-monitoring-best-practices/fig-7.png" width="800px" alt="JobManager memory consumption and garbage collection times." />
+<br />
<i><small>JobManager memory consumption and garbage collection times.</small></i>
</center>
-<br/>
+<p><br /></p>
-**Possible Alerts**
+<p><strong>Possible Alerts</strong></p>
-* `container memory limit` &lt; `container memory + safety margin`
+<ul>
+ <li><code>container memory limit</code> &lt; <code>container memory + safety margin</code></li>
+</ul>
-### CPU
+<h3 id="cpu">CPU</h3>
-Besides memory, you should also monitor the CPU load of the TaskManagers. If
+<p>Besides memory, you should also monitor the CPU load of the TaskManagers. If
your TaskManagers are constantly under very high load, you might be able to
improve the overall performance by decreasing the number of task slots per
TaskManager (in case of a Standalone setup), by providing more resources to the
@@ -5387,48 +5733,61 @@ TaskManager (in case of a containerized setup), or by providing more
TaskManagers. In general, a system already running under very high load during
normal operations, will need much more time to catch-up after recovering from a
downtime. During this time you will see a much higher latency (event-time skew) than
-usual.
+usual.</p>
-A sudden increase in the CPU load might also be attributed to high garbage
-collection pressure, which should be visible in the JVM memory metrics as well.
+<p>A sudden increase in the CPU load might also be attributed to high garbage
+collection pressure, which should be visible in the JVM memory metrics as well.</p>
-If one or a few TaskManagers are constantly under very high load, this can slow
+<p>If one or a few TaskManagers are constantly under very high load, this can slow
down the whole topology due to long checkpoint alignment times and increasing
event-time skew. A common reason is skew in the partition key of the data, which
can be mitigated by pre-aggregating before the shuffle or keying on a more
-evenly distributed key.
+evenly distributed key.</p>
-**Key Metrics**
+<p><strong>Key Metrics</strong></p>
-Metric | Scope | Description |
------- | ----- | ----------- |
-`Status.JVM.CPU.Load` | job-/taskmanager | The recent CPU usage of the JVM. |
+<table>
+ <thead>
+ <tr>
+ <th>Metric</th>
+ <th>Scope</th>
+ <th>Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><code>Status.JVM.CPU.Load</code></td>
+ <td>job-/taskmanager</td>
+ <td>The recent CPU usage of the JVM.</td>
+ </tr>
+ </tbody>
+</table>
-<br/>
+<p><br /></p>
-**Example Dashboard Panel**
+<p><strong>Example Dashboard Panel</strong></p>
<center>
-<img src="{{ site.baseurl }}/img/blog/2019-02-21-monitoring-best-practices/fig-8.png" width="800px" alt="TaskManager & JobManager CPU load."/>
-<br/>
-<i><small>TaskManager & JobManager CPU load.</small></i>
+<img src="/img/blog/2019-02-21-monitoring-best-practices/fig-8.png" width="800px" alt="TaskManager &amp; JobManager CPU load." />
+<br />
+<i><small>TaskManager &amp; JobManager CPU load.</small></i>
</center>
-<br/>
+<p><br /></p>
-## System Resources
+<h2 id="system-resources">System Resources</h2>
-In addition to the JVM metrics above, it is also possible to use Flink’s metrics
-system to gather insights about system resources, i.e. memory, CPU &
+<p>In addition to the JVM metrics above, it is also possible to use Flink’s metrics
+system to gather insights about system resources, i.e. memory, CPU &amp;
network-related metrics for the whole machine as opposed to the Flink processes
alone. System resource monitoring is disabled by default and requires additional
dependencies on the classpath. Please check out the
-[Flink system resource metrics documentation](<{{ site.DOCS_BASE_URL }}flink-docs-release-1.7/monitoring/metrics.html#system-resources>) for
+<a href="https://ci.apache.org/projects/flink/flink-docs-release-1.7/monitoring/metrics.html#system-resources">Flink system resource metrics documentation</a> for
additional guidance and details. System resource monitoring in Flink can be very
-helpful in setups without existing host monitoring capabilities.
+helpful in setups without existing host monitoring capabilities.</p>
-## Conclusion
+<h2 id="conclusion">Conclusion</h2>
-This post tries to shed some light on Flink’s metrics and monitoring system. You
+<p>This post tries to shed some light on Flink’s metrics and monitoring system. You
can utilise it as a starting point when you first think about how to
successfully monitor your Flink application. I highly recommend to start
monitoring your Flink application early on in the development phase. This way
@@ -5436,3138 +5795,3283 @@ you will be able to improve your dashboards and alerts over time and, more
importantly, observe the performance impact of the changes to your application
throughout the development phase. By doing so, you can ask the right questions
about the runtime behaviour of your application, and learn much more about
-Flink’s internals early on.
+Flink’s internals early on.</p>
-Last but not least, this post only scratches the surface of the overall metrics
+<p>Last but not least, this post only scratches the surface of the overall metrics
and monitoring capabilities of Apache Flink. I highly recommend going over
-[Flink’s metrics documentation](<{{ site.DOCS_BASE_URL }}flink-docs-release-1.7/monitoring/metrics.html>)
-for a full reference of Flink’s metrics system.</description>
-<pubDate>Mon, 25 Feb 2019 12:00:00 +0000</pubDate>
+<a href="https://ci.apache.org/projects/flink/flink-docs-release-1.7/monitoring/metrics.html">Flink’s metrics documentation</a>
+for a full reference of Flink’s metrics system.</p>
+</description>
+<pubDate>Mon, 25 Feb 2019 13:00:00 +0100</pubDate>
<link>https://flink.apache.org/news/2019/02/25/monitoring-best-practices.html</link>
<guid isPermaLink="true">/news/2019/02/25/monitoring-best-practices.html</guid>
</item>
<item>
<title>Apache Flink 1.6.4 Released</title>
-<description>The Apache Flink community released the fourth bugfix version of the Apache Flink 1.6 series.
+<description><p>The Apache Flink community released the fourth bugfix version of the Apache Flink 1.6 series.</p>
-This release includes more than 25 fixes and minor improvements for Flink 1.6.3. The list below includes a detailed list of all fixes.
+<p>This release includes more than 25 fixes and minor improvements for Flink 1.6.3. The list below includes a detailed list of all fixes.</p>
-We highly recommend all users to upgrade to Flink 1.6.4.
+<p>We highly recommend all users to upgrade to Flink 1.6.4.</p>
-Updated Maven dependencies:
+<p>Updated Maven dependencies:</p>
-```xml
-<dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-java</artifactId>
- <version>1.6.4</version>
-</dependency>
-<dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-streaming-java_2.11</artifactId>
- <version>1.6.4</version>
-</dependency>
-<dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-clients_2.11</artifactId>
- <version>1.6.4</version>
-</dependency>
-```
+<div class="highlight"><pre><code class="language-xml"><span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-java<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.6.4<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span>
+<span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-streaming-java_2.11<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.6.4<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span>
+<span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-clients_2.11<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.6.4<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span></code></pre></div>
-You can find the binaries on the updated [Downloads page]({{ site.baseurl }}/downloads.html).
+<p>You can find the binaries on the updated <a href="/downloads.html">Downloads page</a>.</p>
-List of resolved issues:
+<p>List of resolved issues:</p>
<h2> Bug
</h2>
<ul>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10721'>FLINK-10721</a>] - Kafka discovery-loop exceptions may be swallowed
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10721">FLINK-10721</a>] - Kafka discovery-loop exceptions may be swallowed
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10761'>FLINK-10761</a>] - MetricGroup#getAllVariables can deadlock
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10761">FLINK-10761</a>] - MetricGroup#getAllVariables can deadlock
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10774'>FLINK-10774</a>] - connection leak when partition discovery is disabled and open throws exception
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10774">FLINK-10774</a>] - connection leak when partition discovery is disabled and open throws exception
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10848'>FLINK-10848</a>] - Flink&#39;s Yarn ResourceManager can allocate too many excess containers
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10848">FLINK-10848</a>] - Flink&#39;s Yarn ResourceManager can allocate too many excess containers
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-11022'>FLINK-11022</a>] - Update LICENSE and NOTICE files for older releases
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11022">FLINK-11022</a>] - Update LICENSE and NOTICE files for older releases
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-11071'>FLINK-11071</a>] - Dynamic proxy classes cannot be resolved when deserializing job graph
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11071">FLINK-11071</a>] - Dynamic proxy classes cannot be resolved when deserializing job graph
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-11084'>FLINK-11084</a>] - Incorrect ouput after two consecutive split and select
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11084">FLINK-11084</a>] - Incorrect ouput after two consecutive split and select
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-11119'>FLINK-11119</a>] - Incorrect Scala example for Table Function
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11119">FLINK-11119</a>] - Incorrect Scala example for Table Function
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-11134'>FLINK-11134</a>] - Invalid REST API request should not log the full exception in Flink logs
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11134">FLINK-11134</a>] - Invalid REST API request should not log the full exception in Flink logs
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-11151'>FLINK-11151</a>] - FileUploadHandler stops working if the upload directory is removed
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11151">FLINK-11151</a>] - FileUploadHandler stops working if the upload directory is removed
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-11173'>FLINK-11173</a>] - Proctime attribute validation throws an incorrect exception message
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11173">FLINK-11173</a>] - Proctime attribute validation throws an incorrect exception message
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-11224'>FLINK-11224</a>] - Log is missing in scala-shell
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11224">FLINK-11224</a>] - Log is missing in scala-shell
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-11232'>FLINK-11232</a>] - Empty Start Time of sub-task on web dashboard
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11232">FLINK-11232</a>] - Empty Start Time of sub-task on web dashboard
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-11234'>FLINK-11234</a>] - ExternalTableCatalogBuilder unable to build a batch-only table
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11234">FLINK-11234</a>] - ExternalTableCatalogBuilder unable to build a batch-only table
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-11235'>FLINK-11235</a>] - Elasticsearch connector leaks threads if no connection could be established
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11235">FLINK-11235</a>] - Elasticsearch connector leaks threads if no connection could be established
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-11251'>FLINK-11251</a>] - Incompatible metric name on prometheus reporter
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11251">FLINK-11251</a>] - Incompatible metric name on prometheus reporter
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-11389'>FLINK-11389</a>] - Incorrectly use job information when call getSerializedTaskInformation in class TaskDeploymentDescriptor
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11389">FLINK-11389</a>] - Incorrectly use job information when call getSerializedTaskInformation in class TaskDeploymentDescriptor
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-11584'>FLINK-11584</a>] - ConfigDocsCompletenessITCase fails DescriptionBuilder#linebreak() is used
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11584">FLINK-11584</a>] - ConfigDocsCompletenessITCase fails DescriptionBuilder#linebreak() is used
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-11585'>FLINK-11585</a>] - Prefix matching in ConfigDocsGenerator can result in wrong assignments
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11585">FLINK-11585</a>] - Prefix matching in ConfigDocsGenerator can result in wrong assignments
</li>
</ul>
-
+
<h2> Improvement
</h2>
<ul>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10910'>FLINK-10910</a>] - Harden Kubernetes e2e test
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10910">FLINK-10910</a>] - Harden Kubernetes e2e test
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-11079'>FLINK-11079</a>] - Skip deployment for flnk-storm-examples
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11079">FLINK-11079</a>] - Skip deployment for flnk-storm-examples
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-11207'>FLINK-11207</a>] - Update Apache commons-compress from 1.4.1 to 1.18
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11207">FLINK-11207</a>] - Update Apache commons-compress from 1.4.1 to 1.18
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-11262'>FLINK-11262</a>] - Bump jython-standalone to 2.7.1
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11262">FLINK-11262</a>] - Bump jython-standalone to 2.7.1
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-11289'>FLINK-11289</a>] - Rework example module structure to account for licensing
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11289">FLINK-11289</a>] - Rework example module structure to account for licensing
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-11304'>FLINK-11304</a>] - Typo in time attributes doc
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11304">FLINK-11304</a>] - Typo in time attributes doc
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-11469'>FLINK-11469</a>] - fix Tuning Checkpoints and Large State doc
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11469">FLINK-11469</a>] - fix Tuning Checkpoints and Large State doc
</li>
</ul>
</description>
-<pubDate>Mon, 25 Feb 2019 00:00:00 +0000</pubDate>
+<pubDate>Mon, 25 Feb 2019 01:00:00 +0100</pubDate>
<link>https://flink.apache.org/news/2019/02/25/release-1.6.4.html</link>
<guid isPermaLink="true">/news/2019/02/25/release-1.6.4.html</guid>
</item>
<item>
<title>Apache Flink 1.7.2 Released</title>
-<description>The Apache Flink community released the second bugfix version of the Apache Flink 1.7 series.
-
-This release includes more than 40 fixes and minor improvements for Flink 1.7.1, covering several critical
-recovery issues as well as problems in the Flink streaming connectors.
-
-The list below includes a detailed list of all fixes.
-We highly recommend all users to upgrade to Flink 1.7.2.
-
-Updated Maven dependencies:
-
-```xml
-<dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-java</artifactId>
- <version>1.7.2</version>
-</dependency>
-<dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-streaming-java_2.11</artifactId>
- <version>1.7.2</version>
-</dependency>
-<dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-clients_2.11</artifactId>
- <version>1.7.2</version>
-</dependency>
-```
-
-You can find the binaries on the updated [Downloads page]({{ site.baseurl }}/downloads.html).
-
-List of resolved issues:
+<description><p>The Apache Flink community released the second bugfix version of the Apache Flink 1.7 series.</p>
+
+<p>This release includes more than 40 fixes and minor improvements for Flink 1.7.1, covering several critical
+recovery issues as well as problems in the Flink streaming connectors.</p>
+
+<p>The list below includes a detailed list of all fixes.
+We highly recommend all users to upgrade to Flink 1.7.2.</p>
+
+<p>Updated Maven dependencies:</p>
+
+<div class="highlight"><pre><code class="language-xml"><span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-java<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.7.2<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span>
+<span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-streaming-java_2.11<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.7.2<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span>
+<span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-clients_2.11<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.7.2<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span></code></pre></div>
+
+<p>You can find the binaries on the updated <a href="/downloads.html">Downloads page</a>.</p>
+
+<p>List of resolved issues:</p>
<h2> Sub-task
</h2>
<ul>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-11179'>FLINK-11179</a>] - JoinCancelingITCase#testCancelSortMatchWhileDoingHeavySorting test error
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11179">FLINK-11179</a>] - JoinCancelingITCase#testCancelSortMatchWhileDoingHeavySorting test error
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-11180'>FLINK-11180</a>] - ProcessFailureCancelingITCase#testCancelingOnProcessFailure
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11180">FLINK-11180</a>] - ProcessFailureCancelingITCase#testCancelingOnProcessFailure
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-11181'>FLINK-11181</a>] - SimpleRecoveryITCaseBase test error
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11181">FLINK-11181</a>] - SimpleRecoveryITCaseBase test error
</li>
</ul>
-
+
<h2> Bug
</h2>
<ul>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10721'>FLINK-10721</a>] - Kafka discovery-loop exceptions may be swallowed
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10721">FLINK-10721</a>] - Kafka discovery-loop exceptions may be swallowed
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10761'>FLINK-10761</a>] - MetricGroup#getAllVariables can deadlock
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10761">FLINK-10761</a>] - MetricGroup#getAllVariables can deadlock
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10774'>FLINK-10774</a>] - connection leak when partition discovery is disabled and open throws exception
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10774">FLINK-10774</a>] - connection leak when partition discovery is disabled and open throws exception
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10848'>FLINK-10848</a>] - Flink&#39;s Yarn ResourceManager can allocate too many excess containers
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10848">FLINK-10848</a>] - Flink&#39;s Yarn ResourceManager can allocate too many excess containers
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-11046'>FLINK-11046</a>] - ElasticSearch6Connector cause thread blocked when index failed with retry
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11046">FLINK-11046</a>] - ElasticSearch6Connector cause thread blocked when index failed with retry
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-11071'>FLINK-11071</a>] - Dynamic proxy classes cannot be resolved when deserializing job graph
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11071">FLINK-11071</a>] - Dynamic proxy classes cannot be resolved when deserializing job graph
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-11083'>FLINK-11083</a>] - CRowSerializerConfigSnapshot is not instantiable
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11083">FLINK-11083</a>] - CRowSerializerConfigSnapshot is not instantiable
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-11084'>FLINK-11084</a>] - Incorrect ouput after two consecutive split and select
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11084">FLINK-11084</a>] - Incorrect ouput after two consecutive split and select
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-11100'>FLINK-11100</a>] - Presto S3 FileSystem E2E test broken
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11100">FLINK-11100</a>] - Presto S3 FileSystem E2E test broken
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-11119'>FLINK-11119</a>] - Incorrect Scala example for Table Function
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11119">FLINK-11119</a>] - Incorrect Scala example for Table Function
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-11134'>FLINK-11134</a>] - Invalid REST API request should not log the full exception in Flink logs
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11134">FLINK-11134</a>] - Invalid REST API request should not log the full exception in Flink logs
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-11145'>FLINK-11145</a>] - Fix Hadoop version handling in binary release script
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11145">FLINK-11145</a>] - Fix Hadoop version handling in binary release script
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-11151'>FLINK-11151</a>] - FileUploadHandler stops working if the upload directory is removed
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11151">FLINK-11151</a>] - FileUploadHandler stops working if the upload directory is removed
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-11168'>FLINK-11168</a>] - LargePlanTest times out on Travis
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11168">FLINK-11168</a>] - LargePlanTest times out on Travis
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-11173'>FLINK-11173</a>] - Proctime attribute validation throws an incorrect exception message
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11173">FLINK-11173</a>] - Proctime attribute validation throws an incorrect exception message
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-11187'>FLINK-11187</a>] - StreamingFileSink with S3 backend transient socket timeout issues
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11187">FLINK-11187</a>] - StreamingFileSink with S3 backend transient socket timeout issues
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-11191'>FLINK-11191</a>] - Exception in code generation when ambiguous columns in MATCH_RECOGNIZE
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11191">FLINK-11191</a>] - Exception in code generation when ambiguous columns in MATCH_RECOGNIZE
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-11194'>FLINK-11194</a>] - missing Scala 2.12 build of HBase connector
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11194">FLINK-11194</a>] - missing Scala 2.12 build of HBase connector
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-11201'>FLINK-11201</a>] - Document SBT dependency requirements when using MiniClusterResource
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11201">FLINK-11201</a>] - Document SBT dependency requirements when using MiniClusterResource
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-11224'>FLINK-11224</a>] - Log is missing in scala-shell
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11224">FLINK-11224</a>] - Log is missing in scala-shell
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-11227'>FLINK-11227</a>] - The DescriptorProperties contains some bounds checking errors
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11227">FLINK-11227</a>] - The DescriptorProperties contains some bounds checking errors
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-11232'>FLINK-11232</a>] - Empty Start Time of sub-task on web dashboard
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11232">FLINK-11232</a>] - Empty Start Time of sub-task on web dashboard
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-11234'>FLINK-11234</a>] - ExternalTableCatalogBuilder unable to build a batch-only table
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11234">FLINK-11234</a>] - ExternalTableCatalogBuilder unable to build a batch-only table
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-11235'>FLINK-11235</a>] - Elasticsearch connector leaks threads if no connection could be established
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11235">FLINK-11235</a>] - Elasticsearch connector leaks threads if no connection could be established
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-11246'>FLINK-11246</a>] - Fix distinct AGG visibility issues
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11246">FLINK-11246</a>] - Fix distinct AGG visibility issues
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-11251'>FLINK-11251</a>] - Incompatible metric name on prometheus reporter
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11251">FLINK-11251</a>] - Incompatible metric name on prometheus reporter
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-11279'>FLINK-11279</a>] - Invalid week interval parsing in ExpressionParser
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11279">FLINK-11279</a>] - Invalid week interval parsing in ExpressionParser
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-11302'>FLINK-11302</a>] - FlinkS3FileSystem uses an incorrect path for temporary files.
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11302">FLINK-11302</a>] - FlinkS3FileSystem uses an incorrect path for temporary files.
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-11389'>FLINK-11389</a>] - Incorrectly use job information when call getSerializedTaskInformation in class TaskDeploymentDescriptor
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11389">FLINK-11389</a>] - Incorrectly use job information when call getSerializedTaskInformation in class TaskDeploymentDescriptor
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-11419'>FLINK-11419</a>] - StreamingFileSink fails to recover after taskmanager failure
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11419">FLINK-11419</a>] - StreamingFileSink fails to recover after taskmanager failure
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-11436'>FLINK-11436</a>] - Java deserialization failure of the AvroSerializer when used in an old CompositeSerializers
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11436">FLINK-11436</a>] - Java deserialization failure of the AvroSerializer when used in an old CompositeSerializers
</li>
</ul>
-
+
<h2> New Feature
</h2>
<ul>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10457'>FLINK-10457</a>] - Support SequenceFile for StreamingFileSink
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10457">FLINK-10457</a>] - Support SequenceFile for StreamingFileSink
</li>
</ul>
-
+
<h2> Improvement
</h2>
<ul>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10910'>FLINK-10910</a>] - Harden Kubernetes e2e test
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10910">FLINK-10910</a>] - Harden Kubernetes e2e test
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-11023'>FLINK-11023</a>] - Update LICENSE and NOTICE files for flink-connectors
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11023">FLINK-11023</a>] - Update LICENSE and NOTICE files for flink-connectors
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-11079'>FLINK-11079</a>] - Skip deployment for flink-storm-examples
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11079">FLINK-11079</a>] - Skip deployment for flink-storm-examples
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-11207'>FLINK-11207</a>] - Update Apache commons-compress from 1.4.1 to 1.18
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11207">FLINK-11207</a>] - Update Apache commons-compress from 1.4.1 to 1.18
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-11216'>FLINK-11216</a>] - Back to top button is missing in the Joining document and is not properly placed in the Process Function document
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11216">FLINK-11216</a>] - Back to top button is missing in the Joining document and is not properly placed in the Process Function document
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-11262'>FLINK-11262</a>] - Bump jython-standalone to 2.7.1
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11262">FLINK-11262</a>] - Bump jython-standalone to 2.7.1
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-11289'>FLINK-11289</a>] - Rework example module structure to account for licensing
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11289">FLINK-11289</a>] - Rework example module structure to account for licensing
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-11304'>FLINK-11304</a>] - Typo in time attributes doc
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11304">FLINK-11304</a>] - Typo in time attributes doc
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-11331'>FLINK-11331</a>] - Fix errors in tableApi.md and functions.md
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11331">FLINK-11331</a>] - Fix errors in tableApi.md and functions.md
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-11469'>FLINK-11469</a>] - fix Tuning Checkpoints and Large State doc
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11469">FLINK-11469</a>] - fix Tuning Checkpoints and Large State doc
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-11473'>FLINK-11473</a>] - Clarify Documenation on Latency Tracking
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11473">FLINK-11473</a>] - Clarify Documenation on Latency Tracking
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-11628'>FLINK-11628</a>] - Cache maven on travis
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11628">FLINK-11628</a>] - Cache maven on travis
</li>
</ul>
</description>
-<pubDate>Fri, 15 Feb 2019 12:00:00 +0000</pubDate>
+<pubDate>Fri, 15 Feb 2019 13:00:00 +0100</pubDate>
<link>https://flink.apache.org/news/2019/02/15/release-1.7.2.html</link>
<guid isPermaLink="true">/news/2019/02/15/release-1.7.2.html</guid>
</item>
<item>
<title>Batch as a Special Case of Streaming and Alibaba's contribution of Blink</title>
-<description>Last week, we [broke the news](https://lists.apache.org/thread.html/2f7330e85d702a53b4a2b361149930b50f2e89d8e8a572f8ee2a0e6d@%3Cdev.flink.apache.org%3E) that Alibaba decided to contribute its Flink-fork, called Blink, back to the Apache Flink project. Why is that a big thing for Flink, what will it mean for users and the community, and how does it fit into Flink’s overall vision? Let's take a step back to understand this better...
+<description><p>Last week, we <a href="https://lists.apache.org/thread.html/2f7330e85d702a53b4a2b361149930b50f2e89d8e8a572f8ee2a0e6d@%3Cdev.flink.apache.org%3E">broke the news</a> that Alibaba decided to contribute its Flink-fork, called Blink, back to the Apache Flink project. Why is that a big thing for Flink, what will it mean for users and the community, and how does it fit into Flink’s overall vision? Let’s take a step back to understand this better…</p>
-## A Unified Approach to Batch and Streaming
+<h2 id="a-unified-approach-to-batch-and-streaming">A Unified Approach to Batch and Streaming</h2>
-Since its early days, Apache Flink has followed the philosophy of taking a unified approach to batch and streaming data processing. The core building block is *"continuous processing of unbounded data streams"*: if you can do that, you can also do offline processing of bounded data sets (batch processing use cases), because these are just streams that happen to end at some point.
+<p>Since its early days, Apache Flink has followed the philosophy of taking a unified approach to batch and streaming data processing. The core building block is <em>“continuous processing of unbounded data streams”</em>: if you can do that, you can also do offline processing of bounded data sets (batch processing use cases), because these are just streams that happen to end at some point.</p>
<center>
-<img src="{{ site.baseurl }}/img/blog/unified-batch-streaming-blink/bounded-unbounded.png" width="600px" alt="Processing of bounded and unbounded data."/>
+<img src="/img/blog/unified-batch-streaming-blink/bounded-unbounded.png" width="600px" alt="Processing of bounded and unbounded data." />
</center>
-<br>
+<p><br /></p>
-The *"streaming first, with batch as a special case of streaming"* philosophy is supported by various projects (for example [Flink](https://flink.apache.org), [Beam](https://beam.apache.org), etc.) and often been cited as a powerful way to build data applications that [generalize across real-time and offline processing](https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-101) and to help greatly reduce the complexity of data infrastructures.
+<p>The <em>“streaming first, with batch as a special case of streaming”</em> philosophy is supported by various projects (for example <a href="https://flink.apache.org">Flink</a>, <a href="https://beam.apache.org">Beam</a>, etc.) and often been cited as a powerful way to build data applications that <a href="https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-101">generalize across real-time and offl [...]
-### Why are there still batch processors?
+<h3 id="why-are-there-still-batch-processors">Why are there still batch processors?</h3>
-However, *"batch is just a special case of streaming"* does not mean that any stream processor is now the right tool for your batch processing use cases - the introduction of stream processors did not render batch processors obsolete:
+<p>However, <em>“batch is just a special case of streaming”</em> does not mean that any stream processor is now the right tool for your batch processing use cases - the introduction of stream processors did not render batch processors obsolete:</p>
-* Pure stream processing systems are very slow at batch processing workloads. No one would consider it a good idea to use a stream processor that shuffles through message queues to analyze large amounts of available data.
-
-* Unified APIs like [Apache Beam](https://beam.apache.org) often delegate to different runtimes depending on whether the data is continuous/unbounded of fix/bounded. For example, the implementations of the batch and streaming runtime of Google Cloud Dataflow are different, to get the desired performance and resilience in each case.
-
-* *Apache Flink* has a streaming API that can do bounded/unbounded use cases, but still offers a separate DataSet API and runtime stack that is faster for batch use cases.
+<ul>
+ <li>
+ <p>Pure stream processing systems are very slow at batch processing workloads. No one would consider it a good idea to use a stream processor that shuffles through message queues to analyze large amounts of available data.</p>
+ </li>
+ <li>
+ <p>Unified APIs like <a href="https://beam.apache.org">Apache Beam</a> often delegate to different runtimes depending on whether the data is continuous/unbounded of fix/bounded. For example, the implementations of the batch and streaming runtime of Google Cloud Dataflow are different, to get the desired performance and resilience in each case.</p>
+ </li>
+ <li>
+ <p><em>Apache Flink</em> has a streaming API that can do bounded/unbounded use cases, but still offers a separate DataSet API and runtime stack that is faster for batch use cases.</p>
+ </li>
+</ul>
-What is the reason for the above? Where did *"batch is just a special case of streaming"* go wrong?
+<p>What is the reason for the above? Where did <em>“batch is just a special case of streaming”</em> go wrong?</p>
-The answer is simple, nothing is wrong with that paradigm. Unifying batch and streaming in the API is one aspect. One needs to also exploit certain characteristics of the special case “bounded data” in the runtime to competitively handle batch processing use cases. After all, batch processors have been built specifically for that special case.
+<p>The answer is simple, nothing is wrong with that paradigm. Unifying batch and streaming in the API is one aspect. One needs to also exploit certain characteristics of the special case “bounded data” in the runtime to competitively handle batch processing use cases. After all, batch processors have been built specifically for that special case.</p>
-## Batch on top of a Streaming Runtime
+<h2 id="batch-on-top-of-a-streaming-runtime">Batch on top of a Streaming Runtime</h2>
-We always believed that it is possible to have a runtime that is state-of-the-art for both stream processing and batch processing use cases at the same time. A runtime that is streaming-first, but can exploit just the right amount of special properties of bounded streams to be as fast for batch use cases as dedicated batch processors. **This is the unique approach that Flink takes.**
+<p>We always believed that it is possible to have a runtime that is state-of-the-art for both stream processing and batch processing use cases at the same time. A runtime that is streaming-first, but can exploit just the right amount of special properties of bounded streams to be as fast for batch use cases as dedicated batch processors. <strong>This is the unique approach that Flink takes.</strong></p>
-Apache Flink has a network stack that supports both [low-latency/high-throughput streaming data exchanges](https://www.ververica.com/flink-forward-berlin/resources/improving-throughput-and-latency-with-flinks-network-stack), as well as high-throughput batch shuffles. Flink has streaming runtime operators for many operations, but also specialized operators for bounded inputs, which get used when you choose the DataSet API or select the batch environment in the Table API.
+<p>Apache Flink has a network stack that supports both <a href="https://www.ververica.com/flink-forward-berlin/resources/improving-throughput-and-latency-with-flinks-network-stack">low-latency/high-throughput streaming data exchanges</a>, as well as high-throughput batch shuffles. Flink has streaming runtime operators for many operations, but also specialized operators for bounded inputs, which get used when you choose the DataSet API or select the batch envir [...]
<center>
-<img src="{{ site.baseurl }}/img/blog/unified-batch-streaming-blink/stream-batch-joins.png" width="500px" alt="Streaming and batch joins."/>
-<br>
+<img src="/img/blog/unified-batch-streaming-blink/stream-batch-joins.png" width="500px" alt="Streaming and batch joins." />
+<br />
<i>The figure illustrates a streaming join and a batch join. The batch join can read one input fully into a hash table and then probe with the other input. The stream join needs to build tables for both sides, because it needs to continuously process both inputs.
For data larger than memory, the batch join can partition both data sets into subsets that fit in memory (data hits disk once) whereas the continuous nature of the stream join requires it to always keep all data in the table and repeatedly hit disk on cache misses.</i>
</center>
-<br>
+<p><br /></p>
-Because of that, Apache Flink has been actually demonstrating some pretty impressive batch processing performance since its early days. The below benchmark is a bit older, but validated our architectural approach early on.
+<p>Because of that, Apache Flink has been actually demonstrating some pretty impressive batch processing performance since its early days. The below benchmark is a bit older, but validated our architectural approach early on.</p>
<center>
-<img src="{{ site.baseurl }}/img/blog/unified-batch-streaming-blink/sort-performance.png" width="500px" alt="Sort performance."/>
-<br>
-<i>Time to sort 3.2 TB (80 GB/node), in seconds<br>
+<img src="/img/blog/unified-batch-streaming-blink/sort-performance.png" width="500px" alt="Sort performance." />
+<br />
+<i>Time to sort 3.2 TB (80 GB/node), in seconds<br />
(<a href="https://www.slideshare.net/FlinkForward/dongwon-kim-a-comparative-performance-evaluation-of-flink" target="blank">Presentation by Dongwon Kim, Flink Forward Berlin 2015</a>.)</i>
</center>
-<br>
-
-## What is still missing?
-
-To conclude the approach and make Flink's experience on bounded data (batch) state-of-the-art, we need to add a few more enhancements. We believe that these features are key to realizing our vision:
-
-**(1) A truly unified runtime operator stack**: Currently the bounded and unbounded operators have a different network and threading model and don't mix and match. The original reason was that batch operators followed a "pull model" (easier for batch algorithms), while streaming operators followed a "push model" (better latency/throughput characteristics). In a unified stack, continuous streaming operators are the foundation. When operating on bounded data without [...]
+<p><br /></p>
-**(2) Exploiting bounded streams to reduce the scope of fault tolerance**: When input data is bounded, it is possible to completely buffer data during shuffles (memory or disk) and replay that data after a failure. This makes recovery more fine grained and thus much more efficient.
+<h2 id="what-is-still-missing">What is still missing?</h2>
-**(3) Exploiting bounded stream operator properties for scheduling**: A continuous unbounded streaming application needs (by definition) all operators running at the same time. An application on bounded data can schedule operations after another, depending on how the operators consume data (e.g., first build hash table, then probe hash table). This increases resource efficiency.
+<p>To conclude the approach and make Flink’s experience on bounded data (batch) state-of-the-art, we need to add a few more enhancements. We believe that these features are key to realizing our vision:</p>
-**(4) Enabling these special case optimizations for the DataStream API**: Currently, only the Table API (which is unified across bounded/unbounded streams) activates these optimizations when working on bounded data.
+<p><strong>(1) A truly unified runtime operator stack</strong>: Currently the bounded and unbounded operators have a different network and threading model and don’t mix and match. The original reason was that batch operators followed a “pull model” (easier for batch algorithms), while streaming operators followed a “push model” (better latency/throughput characteristics). In a unified stack, continuous streaming operators are the foundation. When operating on bounded da [...]
-**(5) Performance and coverage for SQL**: SQL is the de-facto standard data language, and while it is also being rapidly adopted for continuous streaming use cases, there is absolutely no way past it for bounded/batch use cases. To be competitive with the best batch engines, Flink needs more coverage and performance for the SQL query execution. While the core data-plane in Flink is high performance, the speed of SQL execution ultimately depends a lot also on optimizer rules, a rich set o [...]
+<p><strong>(2) Exploiting bounded streams to reduce the scope of fault tolerance</strong>: When input data is bounded, it is possible to completely buffer data during shuffles (memory or disk) and replay that data after a failure. This makes recovery more fine grained and thus much more efficient.</p>
-## Enter Blink
+<p><strong>(3) Exploiting bounded stream operator properties for scheduling</strong>: A continuous unbounded streaming application needs (by definition) all operators running at the same time. An application on bounded data can schedule operations after another, depending on how the operators consume data (e.g., first build hash table, then probe hash table). This increases resource efficiency.</p>
-Blink is a fork of Apache Flink, originally created inside Alibaba to improve Flink’s behavior for internal use cases. Blink adds a series of improvements and integrations (see the [Readme](https://github.com/apache/flink/blob/blink/README.md) for details), many of which fall into the category of improved bounded-data/batch processing and SQL. In fact, of the above list of features for a unified batch/streaming system, Blink implements significant steps forward in all except (4):
+<p><strong>(4) Enabling these special case optimizations for the DataStream API</strong>: Currently, only the Table API (which is unified across bounded/unbounded streams) activates these optimizations when working on bounded data.</p>
-**Unified Stream Operators:** Blink extends the Flink streaming runtime operator model to support selectively reading from different inputs, while keeping the push model for very low latency. This control over the inputs helps to now support algorithms like hybrid hash-joins on the same operator and threading model as continuous symmetric joins through RocksDB. These operators also form the basis for future features like [“Side Inputs”](https://cwiki.apache.org/confluence/display/FLINK/F [...]
+<p><strong>(5) Performance and coverage for SQL</strong>: SQL is the de-facto standard data language, and while it is also being rapidly adopted for continuous streaming use cases, there is absolutely no way past it for bounded/batch use cases. To be competitive with the best batch engines, Flink needs more coverage and performance for the SQL query execution. While the core data-plane in Flink is high performance, the speed of SQL execution ultimately depends a lot als [...]
-**Table API & SQL Query Processor:** The SQL query processor is the component that evolved the changed most compared to the latest Flink master branch:
+<h2 id="enter-blink">Enter Blink</h2>
-- While Flink currently translates queries either into DataSet or DataStream programs (depending on the characteristics of their inputs), Blink translates queries to a data flow of the aforementioned stream operators.
+<p>Blink is a fork of Apache Flink, originally created inside Alibaba to improve Flink’s behavior for internal use cases. Blink adds a series of improvements and integrations (see the <a href="https://github.com/apache/flink/blob/blink/README.md">Readme</a> for details), many of which fall into the category of improved bounded-data/batch processing and SQL. In fact, of the above list of features for a unified batch/streaming system, Blink implements significan [...]
-- Blink adds many more runtime operators for common SQL operations like semi-joins, anti-joins, etc.
+<p><strong>Unified Stream Operators:</strong> Blink extends the Flink streaming runtime operator model to support selectively reading from different inputs, while keeping the push model for very low latency. This control over the inputs helps to now support algorithms like hybrid hash-joins on the same operator and threading model as continuous symmetric joins through RocksDB. These operators also form the basis for future features like <a href="https://cwiki.ap [...]
-- The query planner (optimizer) is still based on Apache Calcite, but has many more optimization rules (incl. join reordering) and uses a proper cost model for planning.
+<p><strong>Table API &amp; SQL Query Processor:</strong> The SQL query processor is the component that evolved the changed most compared to the latest Flink master branch:</p>
-- Stream operators are more aggressively chained.
-
-- The common data structures (sorters, hash tables) and serializers are extended to go even further in operating on binary data and saving serialization overhead. Code generation is used for the row serializers.
+<ul>
+ <li>
+ <p>While Flink currently translates queries either into DataSet or DataStream programs (depending on the characteristics of their inputs), Blink translates queries to a data flow of the aforementioned stream operators.</p>
+ </li>
+ <li>
+ <p>Blink adds many more runtime operators for common SQL operations like semi-joins, anti-joins, etc.</p>
+ </li>
+ <li>
+ <p>The query planner (optimizer) is still based on Apache Calcite, but has many more optimization rules (incl. join reordering) and uses a proper cost model for planning.</p>
+ </li>
+ <li>
+ <p>Stream operators are more aggressively chained.</p>
+ </li>
+ <li>
+ <p>The common data structures (sorters, hash tables) and serializers are extended to go even further in operating on binary data and saving serialization overhead. Code generation is used for the row serializers.</p>
+ </li>
+</ul>
-**Improved Scheduling and Failure Recovery:** Finally, Blink implements several improvements for task scheduling and fault tolerance. The scheduling strategies use resources better by exploiting how the operators process their input data. The failover strategies recover more fine-grained along the boundaries of persistent shuffles. A failed JobManager can be replaced without restarting a running application.
+<p><strong>Improved Scheduling and Failure Recovery:</strong> Finally, Blink implements several improvements for task scheduling and fault tolerance. The scheduling strategies use resources better by exploiting how the operators process their input data. The failover strategies recover more fine-grained along the boundaries of persistent shuffles. A failed JobManager can be replaced without restarting a running application.</p>
-The changes in Blink result in a big improvement in performance. The below numbers were reported by the developers of Blink to give a rough impression of the performance gains.
+<p>The changes in Blink result in a big improvement in performance. The below numbers were reported by the developers of Blink to give a rough impression of the performance gains.</p>
<center>
-<img src="{{ site.baseurl }}/img/blog/unified-batch-streaming-blink/blink-flink-tpch.png" width="600px" alt="TPC-H performance of Blink and Flink."/>
-<br>
-<i>Relative performance of Blink versus Flink 1.6.0 in the TPC-H benchmark, query by query.<br>
-The performance improvement is in average 10x.<br>
+<img src="/img/blog/unified-batch-streaming-blink/blink-flink-tpch.png" width="600px" alt="TPC-H performance of Blink and Flink." />
+<br />
+<i>Relative performance of Blink versus Flink 1.6.0 in the TPC-H benchmark, query by query.<br />
+The performance improvement is in average 10x.<br />
<a href="https://www.ververica.com/flink-forward-berlin/resources/unified-engine-for-data-processing-and-ai" target="blank">Presentation by Xiaowei Jiang at Flink Forward Berlin, 2018</a>.)</i>
</center>
-<br>
+<p><br /></p>
<center>
-<img src="{{ site.baseurl }}/img/blog/unified-batch-streaming-blink/blink-spark-tpcds.png" width="600px" alt="TPC-DS performace of Blink and Spark."/>
-<br>
-<i>Performance of Blink versus Spark in the TPC-DS benchmark, aggregate time for all queries together.<br>
+<img src="/img/blog/unified-batch-streaming-blink/blink-spark-tpcds.png" width="600px" alt="TPC-DS performace of Blink and Spark." />
+<br />
+<i>Performance of Blink versus Spark in the TPC-DS benchmark, aggregate time for all queries together.<br />
<a href="https://www.bilibili.com/video/av42325467/?p=3" target="blank">Presentation by Xiaowei Jiang at Flink Forward Beijing, 2018</a>.</i>
</center>
-<br>
-
-## How do we plan to merge Blink and Flink?
+<p><br /></p>
-Blink’s code is currently available as a [branch](https://github.com/apache/flink/tree/blink) in the Apache Flink repository. It is a challenge to merge a such big amount of changes, while making the merge process as non-disruptive as possible and keeping public APIs as stable as possible.
+<h2 id="how-do-we-plan-to-merge-blink-and-flink">How do we plan to merge Blink and Flink?</h2>
-The community’s [merge plan](https://lists.apache.org/thread.html/6066abd0f09fc1c41190afad67770ede8efd0bebc36f00938eecc118@%3Cdev.flink.apache.org%3E) focuses initially on the bounded/batch processing features mentioned above and follows the following approach to ensure a smooth integration:
+<p>Blink’s code is currently available as a <a href="https://github.com/apache/flink/tree/blink">branch</a> in the Apache Flink repository. It is a challenge to merge a such big amount of changes, while making the merge process as non-disruptive as possible and keeping public APIs as stable as possible.</p>
-* To merge Blink’s _SQL/Table API query processor_ enhancements, we exploit the fact that both Flink and Blink have the same APIs: SQL and the Table API.
-Following some restructuring of the Table/SQL module ([FLIP-32](https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions)) we plan to merge the Blink query planner (optimizer) and runtime (operators) as an additional query processor next to the current SQL runtime. Think of it as two different runners for the same APIs.<br>
-Initially, users will be able to select which query processor to use. After a transition period in which the new query processor will be developed to subsume the current query processor, the current processor will most likely be deprecated and eventually dropped. Given that SQL is such a well defined interface, we anticipate that this transition has little friction for users. Mostly a pleasant surprise to have broader SQL feature coverage and a boost in performance.
+<p>The community’s <a href="https://lists.apache.org/thread.html/6066abd0f09fc1c41190afad67770ede8efd0bebc36f00938eecc118@%3Cdev.flink.apache.org%3E">merge plan</a> focuses initially on the bounded/batch processing features mentioned above and follows the following approach to ensure a smooth integration:</p>
-* To support the merge of Blink’s _enhancements to scheduling and recovery_ for jobs on bounded data, the Flink community is already working on refactoring its current schedule and adding support for [pluggable scheduling and fail-over strategies](https://issues.apache.org/jira/browse/FLINK-10429).<br>
-Once this effort is finished, we can add Blink’s scheduling and recovery strategies as a new scheduling strategy that is used by the new query processor. Eventually, we plan to use the new scheduling strategy also for bounded DataStream programs.
-
-* The extended catalog support, DDL support, as well as support for Hive’s catalog and integrations is currently going through separate design discussions. We plan to leverage existing code here whenever it makes sense.
+<ul>
+ <li>
+ <p>To merge Blink’s <em>SQL/Table API query processor</em> enhancements, we exploit the fact that both Flink and Blink have the same APIs: SQL and the Table API.
+Following some restructuring of the Table/SQL module (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions">FLIP-32</a>) we plan to merge the Blink query planner (optimizer) and runtime (operators) as an additional query processor next to the current SQL runtime. Think of it as two different runners for the same APIs.<br />
+Initially, users will be able to select which query processor to use. After a transition period in which the new query processor will be developed to subsume the current query processor, the current processor will most likely be deprecated and eventually dropped. Given that SQL is such a well defined interface, we anticipate that this transition has little friction for users. Mostly a pleasant surprise to have broader SQL feature coverage and a boost in performance.</p>
+ </li>
+ <li>
+ <p>To support the merge of Blink’s <em>enhancements to scheduling and recovery</em> for jobs on bounded data, the Flink community is already working on refactoring its current schedule and adding support for <a href="https://issues.apache.org/jira/browse/FLINK-10429">pluggable scheduling and fail-over strategies</a>.<br />
+Once this effort is finished, we can add Blink’s scheduling and recovery strategies as a new scheduling strategy that is used by the new query processor. Eventually, we plan to use the new scheduling strategy also for bounded DataStream programs.</p>
+ </li>
+ <li>
+ <p>The extended catalog support, DDL support, as well as support for Hive’s catalog and integrations is currently going through separate design discussions. We plan to leverage existing code here whenever it makes sense.</p>
+ </li>
+</ul>
-## Summary
+<h2 id="summary">Summary</h2>
-We believe that the data processing stack of the future is based on stream processing: The elegance of stream processing with its ability to model offline processing (batch), real-time data processing, and event-driven applications in the same way, while offering high performance and consistency is simply too compelling.
+<p>We believe that the data processing stack of the future is based on stream processing: The elegance of stream processing with its ability to model offline processing (batch), real-time data processing, and event-driven applications in the same way, while offering high performance and consistency is simply too compelling.</p>
-Exploiting certain properties of bounded data is important for a stream processor to achieve the same performance as dedicated batch processors. While Flink always supported batch processing, the project is taking the next step in building a unified runtime and towards **becoming a stream processor that is competitive with batch processing systems even on their home turf: OLAP SQL.** The contribution of Alibaba’s Blink code helps the Flink community to pick up the speed on this developme [...]
-<pubDate>Wed, 13 Feb 2019 12:00:00 +0000</pubDate>
+<p>Exploiting certain properties of bounded data is important for a stream processor to achieve the same performance as dedicated batch processors. While Flink always supported batch processing, the project is taking the next step in building a unified runtime and towards <strong>becoming a stream processor that is competitive with batch processing systems even on their home turf: OLAP SQL.</strong> The contribution of Alibaba’s Blink code helps the Flink community to p [...]
+</description>
+<pubDate>Wed, 13 Feb 2019 13:00:00 +0100</pubDate>
<link>https://flink.apache.org/news/2019/02/13/unified-batch-streaming-blink.html</link>
<guid isPermaLink="true">/news/2019/02/13/unified-batch-streaming-blink.html</guid>
</item>
<item>
<title>Apache Flink 1.5.6 Released</title>
-<description>The Apache Flink community released the sixth and last bugfix version of the Apache Flink 1.5 series.
+<description><p>The Apache Flink community released the sixth and last bugfix version of the Apache Flink 1.5 series.</p>
-This release includes more than 47 fixes and minor improvements for Flink 1.5.5. The list below includes a detailed list of all fixes.
+<p>This release includes more than 47 fixes and minor improvements for Flink 1.5.5. The list below includes a detailed list of all fixes.</p>
-We highly recommend all users to upgrade to Flink 1.5.6.
+<p>We highly recommend all users to upgrade to Flink 1.5.6.</p>
-Updated Maven dependencies:
+<p>Updated Maven dependencies:</p>
-```xml
-<dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-java</artifactId>
- <version>1.5.6</version>
-</dependency>
-<dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-streaming-java_2.11</artifactId>
- <version>1.5.6</version>
-</dependency>
-<dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-clients_2.11</artifactId>
- <version>1.5.6</version>
-</dependency>
-```
+<div class="highlight"><pre><code class="language-xml"><span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-java<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.5.6<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span>
+<span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-streaming-java_2.11<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.5.6<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span>
+<span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-clients_2.11<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.5.6<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span></code></pre></div>
-You can find the binaries on the updated [Downloads page](http://flink.apache.org/downloads.html).
+<p>You can find the binaries on the updated <a href="http://flink.apache.org/downloads.html">Downloads page</a>.</p>
-List of resolved issues:
+<p>List of resolved issues:</p>
<h2> Sub-task
</h2>
<ul>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10252'>FLINK-10252</a>] - Handle oversized metric messages
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10252">FLINK-10252</a>] - Handle oversized metric messages
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10863'>FLINK-10863</a>] - Assign uids to all operators
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10863">FLINK-10863</a>] - Assign uids to all operators
</li>
</ul>
-
+
<h2> Bug
</h2>
<ul>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-8336'>FLINK-8336</a>] - YarnFileStageTestS3ITCase.testRecursiveUploadForYarnS3 test instability
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-8336">FLINK-8336</a>] - YarnFileStageTestS3ITCase.testRecursiveUploadForYarnS3 test instability
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-9646'>FLINK-9646</a>] - ExecutionGraphCoLocationRestartTest.testConstraintsAfterRestart failed on Travis
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-9646">FLINK-9646</a>] - ExecutionGraphCoLocationRestartTest.testConstraintsAfterRestart failed on Travis
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10166'>FLINK-10166</a>] - Dependency problems when executing SQL query in sql-client
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10166">FLINK-10166</a>] - Dependency problems when executing SQL query in sql-client
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10309'>FLINK-10309</a>] - Cancel with savepoint fails with java.net.ConnectException when using the per job-mode
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10309">FLINK-10309</a>] - Cancel with savepoint fails with java.net.ConnectException when using the per job-mode
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10419'>FLINK-10419</a>] - ClassNotFoundException while deserializing user exceptions from checkpointing
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10419">FLINK-10419</a>] - ClassNotFoundException while deserializing user exceptions from checkpointing
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10455'>FLINK-10455</a>] - Potential Kafka producer leak in case of failures
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10455">FLINK-10455</a>] - Potential Kafka producer leak in case of failures
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10482'>FLINK-10482</a>] - java.lang.IllegalArgumentException: Negative number of in progress checkpoints
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10482">FLINK-10482</a>] - java.lang.IllegalArgumentException: Negative number of in progress checkpoints
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10491'>FLINK-10491</a>] - Deadlock during spilling data in SpillableSubpartition
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10491">FLINK-10491</a>] - Deadlock during spilling data in SpillableSubpartition
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10566'>FLINK-10566</a>] - Flink Planning is exponential in the number of stages
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10566">FLINK-10566</a>] - Flink Planning is exponential in the number of stages
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10581'>FLINK-10581</a>] - YarnConfigurationITCase.testFlinkContainerMemory test instability
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10581">FLINK-10581</a>] - YarnConfigurationITCase.testFlinkContainerMemory test instability
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10642'>FLINK-10642</a>] - CodeGen split fields errors when maxGeneratedCodeLength equals 1
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10642">FLINK-10642</a>] - CodeGen split fields errors when maxGeneratedCodeLength equals 1
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10655'>FLINK-10655</a>] - RemoteRpcInvocation not overwriting ObjectInputStream&#39;s ClassNotFoundException
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10655">FLINK-10655</a>] - RemoteRpcInvocation not overwriting ObjectInputStream&#39;s ClassNotFoundException
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10669'>FLINK-10669</a>] - Exceptions &amp; errors are not properly checked in logs in e2e tests
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10669">FLINK-10669</a>] - Exceptions &amp; errors are not properly checked in logs in e2e tests
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10670'>FLINK-10670</a>] - Fix Correlate codegen error
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10670">FLINK-10670</a>] - Fix Correlate codegen error
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10674'>FLINK-10674</a>] - Fix handling of retractions after clean up
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10674">FLINK-10674</a>] - Fix handling of retractions after clean up
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10690'>FLINK-10690</a>] - Tests leak resources via Files.list
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10690">FLINK-10690</a>] - Tests leak resources via Files.list
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10693'>FLINK-10693</a>] - Fix Scala EitherSerializer duplication
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10693">FLINK-10693</a>] - Fix Scala EitherSerializer duplication
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10715'>FLINK-10715</a>] - E2e tests fail with ConcurrentModificationException in MetricRegistryImpl
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10715">FLINK-10715</a>] - E2e tests fail with ConcurrentModificationException in MetricRegistryImpl
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10750'>FLINK-10750</a>] - SocketClientSinkTest.testRetry fails on Travis
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10750">FLINK-10750</a>] - SocketClientSinkTest.testRetry fails on Travis
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10752'>FLINK-10752</a>] - Result of AbstractYarnClusterDescriptor#validateClusterResources is ignored
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10752">FLINK-10752</a>] - Result of AbstractYarnClusterDescriptor#validateClusterResources is ignored
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10753'>FLINK-10753</a>] - Propagate and log snapshotting exceptions
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10753">FLINK-10753</a>] - Propagate and log snapshotting exceptions
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10770'>FLINK-10770</a>] - Some generated functions are not opened properly.
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10770">FLINK-10770</a>] - Some generated functions are not opened properly.
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10773'>FLINK-10773</a>] - Resume externalized checkpoint end-to-end test fails
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10773">FLINK-10773</a>] - Resume externalized checkpoint end-to-end test fails
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10821'>FLINK-10821</a>] - Resuming Externalized Checkpoint E2E test does not resume from Externalized Checkpoint
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10821">FLINK-10821</a>] - Resuming Externalized Checkpoint E2E test does not resume from Externalized Checkpoint
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10839'>FLINK-10839</a>] - Fix implementation of PojoSerializer.duplicate() w.r.t. subclass serializer
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10839">FLINK-10839</a>] - Fix implementation of PojoSerializer.duplicate() w.r.t. subclass serializer
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10856'>FLINK-10856</a>] - Harden resume from externalized checkpoint E2E test
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10856">FLINK-10856</a>] - Harden resume from externalized checkpoint E2E test
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10857'>FLINK-10857</a>] - Conflict between JMX and Prometheus Metrics reporter
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10857">FLINK-10857</a>] - Conflict between JMX and Prometheus Metrics reporter
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10880'>FLINK-10880</a>] - Failover strategies should not be applied to Batch Execution
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10880">FLINK-10880</a>] - Failover strategies should not be applied to Batch Execution
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10913'>FLINK-10913</a>] - ExecutionGraphRestartTest.testRestartAutomatically unstable on Travis
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10913">FLINK-10913</a>] - ExecutionGraphRestartTest.testRestartAutomatically unstable on Travis
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10925'>FLINK-10925</a>] - NPE in PythonPlanStreamer
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10925">FLINK-10925</a>] - NPE in PythonPlanStreamer
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10990'>FLINK-10990</a>] - Enforce minimum timespan in MeterView
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10990">FLINK-10990</a>] - Enforce minimum timespan in MeterView
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10998'>FLINK-10998</a>] - flink-metrics-ganglia has LGPL dependency
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10998">FLINK-10998</a>] - flink-metrics-ganglia has LGPL dependency
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-11011'>FLINK-11011</a>] - Elasticsearch 6 sink end-to-end test unstable
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11011">FLINK-11011</a>] - Elasticsearch 6 sink end-to-end test unstable
</li>
</ul>
-
+
<h2> Improvement
</h2>
<ul>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-4173'>FLINK-4173</a>] - Replace maven-assembly-plugin by maven-shade-plugin in flink-metrics
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4173">FLINK-4173</a>] - Replace maven-assembly-plugin by maven-shade-plugin in flink-metrics
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-9869'>FLINK-9869</a>] - Send PartitionInfo in batch to Improve perfornance
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-9869">FLINK-9869</a>] - Send PartitionInfo in batch to Improve perfornance
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10613'>FLINK-10613</a>] - Remove logger casts in HBaseConnectorITCase
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10613">FLINK-10613</a>] - Remove logger casts in HBaseConnectorITCase
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10614'>FLINK-10614</a>] - Update test_batch_allround.sh e2e to new testing infrastructure
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10614">FLINK-10614</a>] - Update test_batch_allround.sh e2e to new testing infrastructure
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10637'>FLINK-10637</a>] - Start MiniCluster with random REST port
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10637">FLINK-10637</a>] - Start MiniCluster with random REST port
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10678'>FLINK-10678</a>] - Add a switch to run_test to configure if logs should be checked for errors/excepions
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10678">FLINK-10678</a>] - Add a switch to run_test to configure if logs should be checked for errors/excepions
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10906'>FLINK-10906</a>] - docker-entrypoint.sh logs credentails during startup
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10906">FLINK-10906</a>] - docker-entrypoint.sh logs credentails during startup
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10916'>FLINK-10916</a>] - Include duplicated user-specified uid into error message
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10916">FLINK-10916</a>] - Include duplicated user-specified uid into error message
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-11005'>FLINK-11005</a>] - Define flink-sql-client uber-jar dependencies via artifactSet
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11005">FLINK-11005</a>] - Define flink-sql-client uber-jar dependencies via artifactSet
</li>
</ul>
-
+
<h2> Test
</h2>
<ul>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10606'>FLINK-10606</a>] - Construct NetworkEnvironment simple for tests
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10606">FLINK-10606</a>] - Construct NetworkEnvironment simple for tests
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10607'>FLINK-10607</a>] - Unify to remove duplicated NoOpResultPartitionConsumableNotifier
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10607">FLINK-10607</a>] - Unify to remove duplicated NoOpResultPartitionConsumableNotifier
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10827'>FLINK-10827</a>] - Add test for duplicate() to SerializerTestBase
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10827">FLINK-10827</a>] - Add test for duplicate() to SerializerTestBase
</li>
</ul>
</description>
-<pubDate>Wed, 26 Dec 2018 12:00:00 +0000</pubDate>
+<pubDate>Wed, 26 Dec 2018 13:00:00 +0100</pubDate>
<link>https://flink.apache.org/news/2018/12/26/release-1.5.6.html</link>
<guid isPermaLink="true">/news/2018/12/26/release-1.5.6.html</guid>
</item>
<item>
<title>Apache Flink 1.6.3 Released</title>
-<description>The Apache Flink community released the third bugfix version of the Apache Flink 1.6 series.
+<description><p>The Apache Flink community released the third bugfix version of the Apache Flink 1.6 series.</p>
-This release includes more than 80 fixes and minor improvements for Flink 1.6.2. The list below includes a detailed list of all fixes.
+<p>This release includes more than 80 fixes and minor improvements for Flink 1.6.2. The list below includes a detailed list of all fixes.</p>
-We highly recommend all users to upgrade to Flink 1.6.3.
+<p>We highly recommend all users to upgrade to Flink 1.6.3.</p>
-Updated Maven dependencies:
+<p>Updated Maven dependencies:</p>
-```xml
-<dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-java</artifactId>
- <version>1.6.3</version>
-</dependency>
-<dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-streaming-java_2.11</artifactId>
- <version>1.6.3</version>
-</dependency>
-<dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-clients_2.11</artifactId>
- <version>1.6.3</version>
-</dependency>
-```
+<div class="highlight"><pre><code class="language-xml"><span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-java<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.6.3<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span>
+<span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-streaming-java_2.11<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.6.3<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span>
+<span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-clients_2.11<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.6.3<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span></code></pre></div>
-You can find the binaries on the updated [Downloads page](http://flink.apache.org/downloads.html).
+<p>You can find the binaries on the updated <a href="http://flink.apache.org/downloads.html">Downloads page</a>.</p>
-List of resolved issues:
+<p>List of resolved issues:</p>
<h2> Sub-task
</h2>
<ul>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10097'>FLINK-10097</a>] - More tests to increase StreamingFileSink test coverage
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10097">FLINK-10097</a>] - More tests to increase StreamingFileSink test coverage
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10252'>FLINK-10252</a>] - Handle oversized metric messages
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10252">FLINK-10252</a>] - Handle oversized metric messages
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10367'>FLINK-10367</a>] - Avoid recursion stack overflow during releasing SingleInputGate
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10367">FLINK-10367</a>] - Avoid recursion stack overflow during releasing SingleInputGate
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10863'>FLINK-10863</a>] - Assign uids to all operators in general purpose testing job
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10863">FLINK-10863</a>] - Assign uids to all operators in general purpose testing job
</li>
</ul>
-
+
<h2> Bug
</h2>
<ul>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-8336'>FLINK-8336</a>] - YarnFileStageTestS3ITCase.testRecursiveUploadForYarnS3 test instability
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-8336">FLINK-8336</a>] - YarnFileStageTestS3ITCase.testRecursiveUploadForYarnS3 test instability
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-9635'>FLINK-9635</a>] - Local recovery scheduling can cause spread out of tasks
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-9635">FLINK-9635</a>] - Local recovery scheduling can cause spread out of tasks
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-9646'>FLINK-9646</a>] - ExecutionGraphCoLocationRestartTest.testConstraintsAfterRestart failed on Travis
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-9646">FLINK-9646</a>] - ExecutionGraphCoLocationRestartTest.testConstraintsAfterRestart failed on Travis
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-9878'>FLINK-9878</a>] - IO worker threads BLOCKED on SSL Session Cache while CMS full gc
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-9878">FLINK-9878</a>] - IO worker threads BLOCKED on SSL Session Cache while CMS full gc
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10149'>FLINK-10149</a>] - Fink Mesos allocates extra port when not configured to do so.
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10149">FLINK-10149</a>] - Fink Mesos allocates extra port when not configured to do so.
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10166'>FLINK-10166</a>] - Dependency problems when executing SQL query in sql-client
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10166">FLINK-10166</a>] - Dependency problems when executing SQL query in sql-client
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10309'>FLINK-10309</a>] - Cancel with savepoint fails with java.net.ConnectException when using the per job-mode
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10309">FLINK-10309</a>] - Cancel with savepoint fails with java.net.ConnectException when using the per job-mode
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10357'>FLINK-10357</a>] - Streaming File Sink end-to-end test failed with mismatch
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10357">FLINK-10357</a>] - Streaming File Sink end-to-end test failed with mismatch
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10359'>FLINK-10359</a>] - Scala example in DataSet docs is broken
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10359">FLINK-10359</a>] - Scala example in DataSet docs is broken
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10364'>FLINK-10364</a>] - Test instability in NonHAQueryableStateFsBackendITCase#testMapState
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10364">FLINK-10364</a>] - Test instability in NonHAQueryableStateFsBackendITCase#testMapState
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10419'>FLINK-10419</a>] - ClassNotFoundException while deserializing user exceptions from checkpointing
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10419">FLINK-10419</a>] - ClassNotFoundException while deserializing user exceptions from checkpointing
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10425'>FLINK-10425</a>] - taskmanager.host is not respected
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10425">FLINK-10425</a>] - taskmanager.host is not respected
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10455'>FLINK-10455</a>] - Potential Kafka producer leak in case of failures
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10455">FLINK-10455</a>] - Potential Kafka producer leak in case of failures
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10463'>FLINK-10463</a>] - Null literal cannot be properly parsed in Java Table API function call
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10463">FLINK-10463</a>] - Null literal cannot be properly parsed in Java Table API function call
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10481'>FLINK-10481</a>] - Wordcount end-to-end test in docker env unstable
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10481">FLINK-10481</a>] - Wordcount end-to-end test in docker env unstable
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10482'>FLINK-10482</a>] - java.lang.IllegalArgumentException: Negative number of in progress checkpoints
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10482">FLINK-10482</a>] - java.lang.IllegalArgumentException: Negative number of in progress checkpoints
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10491'>FLINK-10491</a>] - Deadlock during spilling data in SpillableSubpartition
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10491">FLINK-10491</a>] - Deadlock during spilling data in SpillableSubpartition
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10566'>FLINK-10566</a>] - Flink Planning is exponential in the number of stages
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10566">FLINK-10566</a>] - Flink Planning is exponential in the number of stages
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10567'>FLINK-10567</a>] - Lost serialize fields when ttl state store with the mutable serializer
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10567">FLINK-10567</a>] - Lost serialize fields when ttl state store with the mutable serializer
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10570'>FLINK-10570</a>] - State grows unbounded when &quot;within&quot; constraint not applied
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10570">FLINK-10570</a>] - State grows unbounded when &quot;within&quot; constraint not applied
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10581'>FLINK-10581</a>] - YarnConfigurationITCase.testFlinkContainerMemory test instability
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10581">FLINK-10581</a>] - YarnConfigurationITCase.testFlinkContainerMemory test instability
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10642'>FLINK-10642</a>] - CodeGen split fields errors when maxGeneratedCodeLength equals 1
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10642">FLINK-10642</a>] - CodeGen split fields errors when maxGeneratedCodeLength equals 1
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10655'>FLINK-10655</a>] - RemoteRpcInvocation not overwriting ObjectInputStream&#39;s ClassNotFoundException
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10655">FLINK-10655</a>] - RemoteRpcInvocation not overwriting ObjectInputStream&#39;s ClassNotFoundException
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10663'>FLINK-10663</a>] - Closing StreamingFileSink can cause NPE
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10663">FLINK-10663</a>] - Closing StreamingFileSink can cause NPE
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10669'>FLINK-10669</a>] - Exceptions &amp; errors are not properly checked in logs in e2e tests
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10669">FLINK-10669</a>] - Exceptions &amp; errors are not properly checked in logs in e2e tests
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10670'>FLINK-10670</a>] - Fix Correlate codegen error
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10670">FLINK-10670</a>] - Fix Correlate codegen error
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10674'>FLINK-10674</a>] - Fix handling of retractions after clean up
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10674">FLINK-10674</a>] - Fix handling of retractions after clean up
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10681'>FLINK-10681</a>] - elasticsearch6.ElasticsearchSinkITCase fails if wrong JNA library installed
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10681">FLINK-10681</a>] - elasticsearch6.ElasticsearchSinkITCase fails if wrong JNA library installed
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10690'>FLINK-10690</a>] - Tests leak resources via Files.list
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10690">FLINK-10690</a>] - Tests leak resources via Files.list
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10693'>FLINK-10693</a>] - Fix Scala EitherSerializer duplication
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10693">FLINK-10693</a>] - Fix Scala EitherSerializer duplication
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10715'>FLINK-10715</a>] - E2e tests fail with ConcurrentModificationException in MetricRegistryImpl
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10715">FLINK-10715</a>] - E2e tests fail with ConcurrentModificationException in MetricRegistryImpl
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10750'>FLINK-10750</a>] - SocketClientSinkTest.testRetry fails on Travis
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10750">FLINK-10750</a>] - SocketClientSinkTest.testRetry fails on Travis
</li>
-<li>[<a href='https://issues.apache.org/jira/browse/FLINK-10752'>FLINK-10752</a>] - Result of AbstractYarnClusterDescriptor#validateClusterResources is ignored
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10752">FLINK-10752</a>] - Result of AbstractYarnClusterDescriptor#validateClusterResources is ignored
... 15232 lines suppressed ...