You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by sj...@apache.org on 2020/07/09 18:23:01 UTC
[flink-web] 02/02: rebuild website
This is an automated email from the ASF dual-hosted git repository.
sjwiesman pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/flink-web.git
commit 0ae6413a7f14fcdde05c3c3e6e2219fbc29f9696
Author: Seth Wiesman <sj...@gmail.com>
AuthorDate: Thu Jul 9 13:22:42 2020 -0500
rebuild website
---
content/blog/feed.xml | 16781 ++++++++++++++++++++++++++
content/news/2020/07/06/release-1.11.0.html | 594 +
2 files changed, 17375 insertions(+)
diff --git a/content/blog/feed.xml b/content/blog/feed.xml
new file mode 100644
index 0000000..6da44a0
--- /dev/null
+++ b/content/blog/feed.xml
@@ -0,0 +1,16781 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
+<channel>
+<title>Flink Blog Feed</title>
+<description>Flink Blog</description>
+<link>https://flink.apache.org/blog</link>
+<atom:link href="https://flink.apache.org/blog/feed.xml" rel="self" type="application/rss+xml" />
+
+<item>
+<title>Apache Flink 1.11.0 Release Announcement</title>
+<description><p>The Apache Flink community is proud to announce the release of Flink 1.11.0! More than 200 contributors worked on over 1.3k issues to bring significant improvements to usability as well as new features to Flink users across the whole API stack. Some highlights that we’re particularly excited about are:</p>
+
+<ul>
+ <li>
+ <p>The core engine is introducing <strong>unaligned checkpoints</strong>, a major change to Flink’s fault tolerance mechanism that improves checkpointing performance under heavy backpressure.</p>
+ </li>
+ <li>
+ <p>A <strong>new Source API</strong> that simplifies the implementation of (custom) sources by unifying batch and streaming execution, as well as offloading internals such as event-time handling, watermark generation or idleness detection to Flink.</p>
+ </li>
+ <li>
+ <p>Flink SQL is introducing <strong>Support for Change Data Capture (CDC)</strong> to easily consume and interpret database changelogs from tools like Debezium. The renewed <strong>FileSystem Connector</strong> also expands the set of use cases and formats supported in the Table API/SQL, enabling scenarios like streaming data directly from Kafka to Hive.</p>
+ </li>
+ <li>
+ <p>Multiple performance optimizations to PyFlink, including support for <strong>vectorized User-defined Functions (Pandas UDFs)</strong>. This improves interoperability with libraries like Pandas and NumPy, making Flink more powerful for data science and ML workloads.</p>
+ </li>
+</ul>
+
+<p>Read on for all major new features and improvements, important changes to be aware of and what to expect moving forward!</p>
+
+<div class="page-toc">
+<ul id="markdown-toc">
+ <li><a href="#new-features-and-improvements" id="markdown-toc-new-features-and-improvements">New Features and Improvements</a> <ul>
+ <li><a href="#unaligned-checkpoints-beta" id="markdown-toc-unaligned-checkpoints-beta">Unaligned Checkpoints (Beta)</a></li>
+ <li><a href="#unified-watermark-generators" id="markdown-toc-unified-watermark-generators">Unified Watermark Generators</a></li>
+ <li><a href="#new-data-source-api-beta" id="markdown-toc-new-data-source-api-beta">New Data Source API (Beta)</a></li>
+ <li><a href="#application-mode-deployments" id="markdown-toc-application-mode-deployments">Application Mode Deployments</a></li>
+ <li><a href="#other-improvements" id="markdown-toc-other-improvements">Other Improvements</a></li>
+ <li><a href="#table-apisql-support-for-change-data-capture-cdc" id="markdown-toc-table-apisql-support-for-change-data-capture-cdc">Table API/SQL: Support for Change Data Capture (CDC)</a></li>
+ <li><a href="#table-apisql-jdbc-catalog-interface-and-postgres-catalog" id="markdown-toc-table-apisql-jdbc-catalog-interface-and-postgres-catalog">Table API/SQL: JDBC Catalog Interface and Postgres Catalog</a></li>
+ <li><a href="#table-apisql-filesystem-connector-with-support-for-avro-orc-and-parquet" id="markdown-toc-table-apisql-filesystem-connector-with-support-for-avro-orc-and-parquet">Table API/SQL: FileSystem Connector with Support for Avro, ORC and Parquet</a></li>
+ <li><a href="#table-apisql-support-for-python-udfs" id="markdown-toc-table-apisql-support-for-python-udfs">Table API/SQL: Support for Python UDFs</a></li>
+ <li><a href="#other-improvements-to-the-table-apisql" id="markdown-toc-other-improvements-to-the-table-apisql">Other Improvements to the Table API/SQL</a></li>
+ <li><a href="#pyflink-support-for-pandas-udfs" id="markdown-toc-pyflink-support-for-pandas-udfs">PyFlink: Support for Pandas UDFs</a></li>
+ <li><a href="#other-improvements-to-pyflink" id="markdown-toc-other-improvements-to-pyflink">Other Improvements to PyFlink</a></li>
+ </ul>
+ </li>
+ <li><a href="#important-changes" id="markdown-toc-important-changes">Important Changes</a></li>
+ <li><a href="#release-notes" id="markdown-toc-release-notes">Release Notes</a></li>
+ <li><a href="#list-of-contributors" id="markdown-toc-list-of-contributors">List of Contributors</a></li>
+</ul>
+
+</div>
+
+<p>The binary distribution and source artifacts are now available on the updated <a href="/downloads.html">Downloads page</a> of the Flink website, and the most recent distribution of PyFlink is available on <a href="https://pypi.org/project/apache-flink/">PyPI</a>. Please review the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.11/release-notes/flink-1.11.html">release notes</a> carefully, and c [...]
+
+<p>We encourage you to download the release and share your feedback with the community through the <a href="https://flink.apache.org/community.html#mailing-lists">Flink mailing lists</a> or <a href="https://issues.apache.org/jira/projects/FLINK/summary">JIRA</a>.</p>
+
+<h2 id="new-features-and-improvements">New Features and Improvements</h2>
+
+<h3 id="unaligned-checkpoints-beta">Unaligned Checkpoints (Beta)</h3>
+
+<p>Triggering a checkpoint in Flink will cause a <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.11/internals/stream_checkpointing.html#barriers">checkpoint barrier</a> to flow from the sources of your topology all the way towards the sinks. For operators that receive more than one input stream, the barriers flowing through each channel need to be aligned before the operator can snapshot its state and forward the checkpoint barrier — typical [...]
+
+<ul>
+ <li>
+ <p>Checkpoint barriers will flow much slower through backpressured channels, effectively blocking the remaining channels and their upstream operators during checkpointing;</p>
+ </li>
+ <li>
+ <p>Slow checkpoint barrier propagation leads to longer checkpointing times and can, worst case, result in little to no progress in the application.</p>
+ </li>
+</ul>
+
+<p>To improve the performance of checkpointing under backpressure scenarios, the community is rolling out the first iteration of unaligned checkpoints (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-76%3A+Unaligned+Checkpoints">FLIP-76</a>) with Flink 1.11. Compared to the original checkpointing mechanism (Fig. 1), this approach doesn’t wait for barrier alignment across input channels, instead allowing barriers to overtake in-flight records (i [...]
+
+<div style="line-height:60%;">
+ <br />
+</div>
+
+<div class="row">
+ <div class="col-lg-6">
+ <div class="text-center">
+ <figure>
+ <img src="/img/blog/2020-07-06-release-1.11.0/image1.gif" width="600px" alt="Aligned Checkpoints" />
+ <br /><br />
+ <figcaption><i><b>Fig.1:</b> Aligned Checkpoints</i></figcaption>
+ </figure>
+ </div>
+ </div>
+ <div class="col-lg-6">
+ <div class="text-center">
+ <figure>
+ <img src="/img/blog/2020-07-06-release-1.11.0/image2.png" width="600px" alt="Unaligned Checkpoints" />
+ <br /><br />
+ <figcaption><i><b>Fig.2:</b> Unaligned Checkpoints</i></figcaption>
+ </figure>
+ </div>
+ </div>
+</div>
+
+<div style="line-height:150%;">
+ <br />
+</div>
+
+<p>Because in-flight records have to be persisted as part of the snapshot, unaligned checkpoints will lead to increased checkpoints sizes. On the upside, <strong>checkpointing times are heavily reduced</strong>, so users will see more progress (even in unstable environments) as more up-to-date checkpoints will lighten the recovery process. You can learn more about the current limitations of unaligned checkpoints in the <a href="https://ci.apache.org/projects/fli [...]
+
+<p>As with any beta feature, we appreciate early feedback that you might want to share with the community after giving unaligned checkpoints a try!</p>
+
+<p><span class="label label-info">Info</span> To enable this feature, you need to configure the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.11/api/java/org/apache/flink/streaming/api/environment/CheckpointConfig.html"><code>enableUnalignedCheckpoints</code></a> option in your <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/stream/state/checkpointing.html#enabling-and-con [...]
+
+<h3 id="unified-watermark-generators">Unified Watermark Generators</h3>
+
+<p>So far, watermark generation (prev. also called <em>assignment</em>) relied on two different interfaces: <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.11/api/java/org/apache/flink/streaming/api/functions/AssignerWithPunctuatedWatermarks.html"><code>AssignerWithPunctuatedWatermarks</code></a> and <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.11/api/java/org/apache/flink/streaming/api/f [...]
+
+<p>This gives users more control over watermark emission and simplifies the implementation of new connectors that need to support watermark assignment and timestamp extraction at the source (see <em><a href="#new-data-source-api-beta">New Data Source API</a></em>). Multiple <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.11//dev/event_timestamps_watermarks.html#introduction-to-watermark-strategies">strategies for [...]
+
+<p><strong>Support for Watermark Idleness Detection</strong></p>
+
+<p>The <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.11/api/java/org/apache/flink/api/common/eventtime/WatermarkStrategy.html#withIdleness-java.time.Duration-"><code>WatermarkStrategy.withIdleness()</code></a> method allows you to mark a stream as idle if no events arrive within a configured time (i.e. a timeout duration), which in turn allows handling event time skew properly and preventing idle partitions from holding back th [...]
+
+<p><span class="label label-info">Note</span> <a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-126%3A+Unify+%28and+separate%29+Watermark+Assigners">FLIP-126</a> introduces no breaking changes, but we recommend that users give preference to the new <code>WatermarkGenerator</code> interface moving forward, in preparation for the deprecation of the legacy watermark assigners in future releases.</p>
+
+<h3 id="new-data-source-api-beta">New Data Source API (Beta)</h3>
+
+<p>Up to this point, writing a production-grade source connector for Flink was a non-trivial task that required users to be somewhat familiar with Flink internals and account for implementation details like event time assignment, watermark generation or idleness detection in their code. Flink 1.11 introduces a new Data Source API (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface">FLIP-27</a>) to overcome these lim [...]
+
+<center>
+ <figure>
+ <img src="/img/blog/2020-07-06-release-1.11.0/image3.png" width="600px" alt="Data Source API" />
+ </figure>
+</center>
+
+<div style="line-height:150%;">
+ <br />
+</div>
+
+<p>Separating the work of split discovery and the actual reading of the consumed data (i.e. the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/stream/sources.html#data-source-concepts"><em>splits</em></a>) in different components — resp. the <code>SplitEnumerator</code> and <code>SourceReader</code> — allows mixing and matching different enumeration strategies and split readers.</p>
+
+<p>As an example, the existing Kafka connector has multiple strategies for partition discovery that are intermingled with the rest of the code. With the new interfaces in place, it would only need a single reader implementation and there could be several split enumerators for the different partition discovery strategies.</p>
+
+<p><strong>Batch and Streaming Unification</strong></p>
+
+<p>Source connectors implemented using the Data Source API will be able to work both as a bounded (<em>batch</em>) and unbounded (<em>streaming</em>) source. The difference between both cases is minimal: for bounded input, the <code>SplitEnumerator</code> will generate a fixed set of splits and each split is finite; for unbounded input, either the splits are not finite or the <code>SplitEnumerator</code> keeps generating new splits.&l [...]
+
+<p><strong>Implicit Watermark and Event Time Handling</strong></p>
+
+<p>The <code>TimestampAssigner</code> and <code>WatermarkGenerator</code> run transparently as part of the <code>SourceReader</code> component, so users also don’t have to implement any timestamp extraction or watermark generation code.</p>
+
+<p><span class="label label-info">Note</span> The existing source connectors have not yet been reimplemented using the Data Source API — this is planned for upcoming releases. If you’re looking to implement a new source, please refer to the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/stream/sources.html#data-sources">Data Source documentation</a> and <a href="https://ci.apache.org/projects/flink/flink [...]
+
+<h3 id="application-mode-deployments">Application Mode Deployments</h3>
+
+<p>Prior to Flink 1.11, jobs in a Flink application could either be submitted to a long-running <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.11/concepts/flink-architecture.html#flink-session-cluster">Flink Session Cluster</a> (<em>session mode</em>) or a dedicated <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.11/concepts/flink-architecture.html#flink-job-cluster">Flink Job Cluster</a> [...]
+
+<p>From this release on, Flink gets an additional deployment mode: <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/deployment/#application-mode">Application Mode</a> (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-85+Flink+Application+Mode">FLIP-85</a>); where the <code>main()</code> method runs on the cluster, rather than the client. The job submission becomes a one-step process: you [...]
+
+<p>In Flink 1.11, the community worked to already support <em>application mode</em> in Kubernetes (<a href="https://issues.apache.org/jira/browse/FLINK-10934">FLINK-10934</a>).</p>
+
+<h3 id="other-improvements">Other Improvements</h3>
+
+<p><strong>Unified Memory Configuration for JobManagers (<a href="https://jira.apache.org/jira/browse/FLINK-16614">FLIP-116</a>)</strong></p>
+
+<p>Following the work started in Flink 1.10 to improve memory management and configuration, this release introduces a new memory model that aligns the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/memory/mem_setup_jobmanager.html">JobManagers’ configuration options</a> and terminology with that introduced in <a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors&q [...]
+
+<p><span class="label label-danger">Attention</span> Reusing a previous Flink configuration without any adjustments can result in differently computed memory parameters for the JVM and, as a result, performance changes or even failures. Make sure to check the <a href="https://ci.apache.org/projects/flink/flink-docs-master/ops/memory/mem_migration.html#migrate-job-manager-memory-configuration">migration guide</a> if you’re planning to upda [...]
+
+<p><strong>Improvements to the Flink WebUI (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-75%3A+Flink+Web+UI+Improvement+Proposal">FLIP-75</a>)</strong></p>
+
+<p>In Flink 1.11, the community kicked off a series of improvements to the Flink WebUI. The first to roll out are better TaskManager and JobManager log display (<a href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=147427143">FLIP-103</a>), as well as a new thread dump utility (<a href="https://issues.apache.org/jira/browse/FLINK-14816">FLINK-14816</a>). Some additional work planned for upcoming releases includes bette [...]
+
+<p><strong>Docker Image Unification (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-111%3A+Docker+image+unification">FLIP-111</a>)</strong></p>
+
+<p>With this release, all Docker-related resources have been consolidated into <a href="https://github.com/apache/flink-docker">apache/flink-docker</a> and the entry point script has been extended to allow users to run the default Docker image in different modes without the need to create a custom image. The <a href="https://ci.apache.org/projects/flink/flink-docs-master/ops/deployment/docker.html#customize-flink-image">updated documentation< [...]
+
+<hr />
+
+<h3 id="table-apisql-support-for-change-data-capture-cdc">Table API/SQL: Support for Change Data Capture (CDC)</h3>
+
+<p>Change Data Capture (CDC) has become a popular pattern to capture committed changes from a database and propagate those changes to downstream consumers, for example to keep multiple datastores in sync and avoid common pitfalls such as <a href="https://thorben-janssen.com/dual-writes/">dual writes</a>. Being able to easily ingest and interpret these changelogs into the Table API/SQL has been a highly demanded feature in the Flink community — and it’s now pos [...]
+
+<p>To extend the scope of the Table API/SQL to use cases like CDC, Flink 1.11 introduces new table source and sink interfaces with <strong>changelog mode</strong> (see <em><a href="#other-improvements-to-the-table-apisql">New TableSource and TableSink Interfaces</a></em>) and support for the <a href="https://debezium.io/">Debezium</a> and <a href="https://github.com/alibaba/canal">Canal</a> fo [...]
+
+<center>
+ <figure>
+ <img src="/img/blog/2020-07-06-release-1.11.0/image4.png" width="500px" alt="CDC" />
+ </figure>
+</center>
+
+<div style="line-height:150%;">
+ <br />
+</div>
+
+<p>Users have to specify either <code>“format=debezium-json”</code> or <code>“format=canal-json”</code> in their <code>CREATE TABLE</code> statement to consume changelogs using SQL DDL.</p>
+
+<div class="highlight"><pre><code class="language-sql"><span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">my_table</span> <span class="p">(</span>
+ <span class="p">...</span>
+<span class="p">)</span> <span class="k">WITH</span> <span class="p">(</span>
+ <span class="s1">&#39;connector&#39;</span><span class="o">=</span><span class="s1">&#39;...&#39;</span><span class="p">,</span> <span class="c1">-- e.g. &#39;kafka&#39;</span>
+ <span class="s1">&#39;format&#39;</span><span class="o">=</span><span class="s1">&#39;debezium-json&#39;</span><span class="p">,</span>
+ <span class="s1">&#39;debezium-json.schema-include&#39;</span><span class="o">=</span><span class="s1">&#39;true&#39;</span> <span class="c1">-- default: false (Debezium can be configured to include or exclude the message schema)</span>
+ <span class="s1">&#39;debezium-json.ignore-parse-errors&#39;</span><span class="o">=</span><span class="s1">&#39;true&#39;</span> <span class="c1">-- default: false</span>
+<span class="p">);</span></code></pre></div>
+
+<p>Flink 1.11 only supports Kafka as a changelog source out-of-the-box and JSON-encoded changelogs, with Avro (Debezium) and Protobuf (Canal) planned for future releases. There are also plans to support MySQL binlogs and Kafka compacted topics as sources, as well as to extend changelog support to batch execution.</p>
+
+<p><span class="label label-danger">Attention</span> There is a known issue (<a href="https://issues.apache.org/jira/browse/FLINK-18461">FLINK-18461</a>) that prevents changelog sources from being used to write to upsert sinks (e.g. MySQL, HBase, Elasticsearch). This will be fixed in the next patch release (1.11.1).</p>
+
+<h3 id="table-apisql-jdbc-catalog-interface-and-postgres-catalog">Table API/SQL: JDBC Catalog Interface and Postgres Catalog</h3>
+
+<p>Flink 1.11 introduces a generic JDBC catalog interface (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-93%3A+JDBC+catalog+and+Postgres+catalog">FLIP-93</a>) that enables users of the Table API/SQL to <strong>derive table schemas automatically</strong> from connections to relational databases over <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/table/connect.html#jdbc-connector">JDBC</ [...]
+
+<p>The first implementation, rolling out with the new release, is the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/table/catalogs.html#postgrescatalog">Postgres catalog</a>.</p>
+
+<h3 id="table-apisql-filesystem-connector-with-support-for-avro-orc-and-parquet">Table API/SQL: FileSystem Connector with Support for Avro, ORC and Parquet</h3>
+
+<p>To improve the user experience for end-to-end streaming ETL use cases, the Flink community worked on a new FileSystem Connector for the Table API/SQL (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-115%3A+Filesystem+connector+in+Table">FLIP-115</a>). The implementation is based on Flink’s <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/filesystems/index.html">FileSystem abstraction</a> and reu [...]
+
+<p>This also means that Table API/SQL users can now make use of all formats already supported by StreamingFileSink, like (Avro) Parquet, as well as the new formats introduced with this release, like Avro (<a href="https://issues.apache.org/jira/browse/FLINK-11395">FLINK-11395</a>) and Orc (<a href="https://issues.apache.org/jira/browse/FLINK-10114">FLINK-10114</a>).</p>
+
+<div class="highlight"><pre><code class="language-sql"><span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">my_table</span> <span class="p">(</span>
+ <span class="n">column_name1</span> <span class="nb">INT</span><span class="p">,</span>
+ <span class="n">column_name2</span> <span class="n">STRING</span><span class="p">,</span>
+ <span class="p">...</span>
+ <span class="n">part_name1</span> <span class="nb">INT</span><span class="p">,</span>
+ <span class="n">part_name2</span> <span class="n">STRING</span>
+<span class="p">)</span> <span class="n">PARTITIONED</span> <span class="k">BY</span> <span class="p">(</span><span class="n">part_name1</span><span class="p">,</span> <span class="n">part_name2</span><span class="p">)</span> <span class="k">WITH</span> <span class="p">(</span>
+ <span class="s1">&#39;connector&#39;</span> <span class="o">=</span> <span class="s1">&#39;filesystem&#39;</span><span class="p">,</span>
+ <span class="s1">&#39;path&#39;</span> <span class="o">=</span> <span class="s1">&#39;file:///path/to/file,</span>
+<span class="s1"> &#39;</span><span class="n">format</span><span class="s1">&#39; = &#39;</span><span class="p">...</span><span class="err">&#39;</span><span class="p">,</span> <span class="c1">-- supported formats: Avro, ORC, Parquet, CSV, JSON </span>
+ <span class="p">...</span>
+<span class="p">);</span></code></pre></div>
+
+<p>The new all-rounder FileSystem Connector transparently handles batch and streaming execution, provides exactly-once guarantees and has full partition support, greatly expanding the scope of usage of the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/table/connect.html#file-system-connector">legacy connector</a>. This allows users to easily implement common use cases like <strong>directly streaming data from Kafka to Hive< [...]
+
+<p>You can track the upcoming improvements to the FileSystem Connector in <a href="https://issues.apache.org/jira/browse/FLINK-17778">FLINK-17778</a>.</p>
+
+<h3 id="table-apisql-support-for-python-udfs">Table API/SQL: Support for Python UDFs</h3>
+
+<p>Prior to this release, users of the Table API/SQL were limited to defining UDFs in either Java or Scala. In Flink 1.11, the community worked on expanding the usage scope of the Python language beyond PyFlink and providing support for Python UDFs in the SQL DDL syntax (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-106%3A+Support+Python+UDF+in+SQL+Function+DDL">FLIP-106</a>), as well as the SQL Client (<a href="https://cwiki.apache.o [...]
+
+<h3 id="other-improvements-to-the-table-apisql">Other Improvements to the Table API/SQL</h3>
+
+<p><strong>DDL and DML Compatibility for the Hive Connector (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-123%3A+DDL+and+DML+compatibility+for+Hive+connector">FLIP-123</a>)</strong></p>
+
+<p>Starting from Flink 1.11, users can write SQL statements directly using Hive syntax (HiveQL) in the Table API/SQL and the SQL Client. For this purpose, an additional dialect was introduced and users can now dynamically switch between Flink (<code>default</code>) and Hive (<code>hive</code>) on a per-statement basis. For a complete list of supported DDL and DML statements, check the Hive dialect <a href="https://ci.apache.org/projects/flink/flink-d [...]
+
+<p><strong>Extensions and Improvements to the Flink SQL Syntax</strong></p>
+
+<ul>
+ <li>
+ <p>Flink 1.11 introduces the concept of <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/table/sql/create.html#create-table">primary key constraints</a> to leverage runtime optimizations in Flink SQL DDL (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP+87%3A+Primary+key+constraints+in+Table+API">FLIP-87</a>);</p>
+ </li>
+ <li>
+ <p>View objects are now fully supported in SQL DDL using the <code>CREATE</code>/<code>ALTER</code>/<code>DROP VIEW</code> statements (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-71%3A+E2E+View+support+in+FLINK+SQL">FLIP-71</a>);</p>
+ </li>
+ <li>
+ <p>Users can now specify or override table options in their DQL/DML statements using <a href="https://ci.apache.org/projects/flink/flink-docs-master/dev/table/sql/hints.html#dynamic-table-options">dynamic table options</a> (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-113%3A+Supports+Dynamic+Table+Options+for+Flink+SQL">FLIP-113</a>).</p>
+ </li>
+ <li>
+ <p>To make connector properties less verbose and improve exception handling, some key properties have been refactored (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-122%3A+New+Connector+Property+Keys+for+New+Factory">FLIP-122</a>). This change does not break compatibility, so users can still use the old property keys.</p>
+ </li>
+</ul>
+
+<p><strong>New TableSource and TableSink Interfaces (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-95%3A+New+TableSource+and+TableSink+interfaces">FLIP-95</a>)</strong></p>
+
+<p>Flink 1.11 introduces new table source and sink interfaces (resp. <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.11/api/java/org/apache/flink/table/connector/source/DynamicTableSource.html"><code>DynamicTableSource</code></a> and <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.11/api/java/org/apache/flink/table/connector/sink/DynamicTableSink.html"><code>DynamicTableSink</code>&l [...]
+
+<p><span class="label label-info">Note</span> Although compatibility is not immediately affected, we recommend that Table API/SQL users update any sources and sinks to the new interface stack.</p>
+
+<p><strong>Refactored TableEnvironment Interface (<a href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=134745878">FLIP-84</a>)</strong></p>
+
+<p>The semantics to describe similar behaviours in the <code>TableEnvironment</code> and <code>Table</code> interfaces have diverged over time, leading to an inconsistent and sometimes unclear user experience. To improve this and make programming more fluent in the Table API/SQL, Flink 1.11 introduces new methods that unify behaviours like execution triggering (e.g. <code>executeSql()</code>) and result representation (e.g. <a href="https [...]
+
+<p><span class="label label-info">Note</span> The methods deprecated with FLIP-84 will not be immediately removed, but we recommend that users adopt the newly introduced methods. For a complete list of new and deprecated methods, check the “Summary” section of <a href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=134745878">FLIP-84</a>.</p>
+
+<p><strong>New Type Inference for Table API UDFs (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-65%3A+New+type+inference+for+Table+API+UDFs">FLIP-65</a>)</strong></p>
+
+<p>In Flink 1.9, the community started working on a <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/table/types.html#data-types">new data type system</a> for the Table API to improve its compliance with the SQL standard (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-37%3A+Rework+of+the+Table+API+Type+System">FLIP-37</a>). This work is now close to being completed in Flink 1.11, with the exposure [...]
+
+<hr />
+
+<h3 id="pyflink-support-for-pandas-udfs">PyFlink: Support for Pandas UDFs</h3>
+
+<p>Up to this release, Python UDFs in PyFlink only supported scalar values of standard Python types. This presented some limitations:</p>
+
+<ul>
+ <li>
+ <p>High serialization/deserialization overhead in the process of transferring data between the JVM and the Python processes;</p>
+ </li>
+ <li>
+ <p>Hard to integrate with common Python libraries for high-performance numerical processing like pandas and NumPy.</p>
+ </li>
+</ul>
+
+<p>To overcome these limitations, the community introduced support for (scalar) <strong>vectorized Python UDFs</strong> based on <a href="https://pandas.pydata.org/pandas-docs/stable/getting_started/overview.html">pandas</a> in Flink 1.11 (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-97%3A+Support+Scalar+Vectorized+Python+UDF+in+PyFlink">FLIP-97</a>). The performance of vectorized UDFs is usually much higher [...]
+
+<div class="highlight"><pre><code class="language-python"><span class="nd">@udf</span><span class="p">(</span><span class="n">input_types</span><span class="o">=</span><span class="p">[</span><span class="n">DataTypes</span><span class="o">.</span><span class="n">BIGINT</span>& [...]
+<span class="k">def</span> <span class="nf">add</span><span class="p">(</span><span class="n">i</span><span class="p">,</span> <span class="n">j</span><span class="p">):</span>
+ <span class="k">return</span> <span class="n">i</span> <span class="o">+</span> <span class="n">j</span></code></pre></div>
+
+<p>To mark a UDF as a Pandas UDF, you only need to add an extra parameter <code>udf_type=”pandas”</code> in the udf decorator, as described in the <a href="https://ci.apache.org/projects/flink/flink-docs-master/dev/table/python/vectorized_python_udfs.html#vectorized-user-defined-functions">documentation</a>.</p>
+
+<h3 id="other-improvements-to-pyflink">Other Improvements to PyFlink</h3>
+
+<p><strong>Conversion fromPandas/toPandas (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-120%3A+Support+conversion+between+PyFlink+Table+and+Pandas+DataFrame">FLIP-120</a>)</strong></p>
+
+<p>Arrow is also supported as an optimization to convert between PyFlink tables and <a href="https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html"><code>pandas.DataFrames</code></a>, enabling users to switch processing engines seamlessly without the need for an intermediate connector. For examples on how to use the new <code>fromPandas()</code> and <code>toPandas()</code> methods in PyFlink, check [...]
+
+<p><strong>Support for User-defined Table Functions (UDTFs) (<a href="https://jira.apache.org/jira/browse/FLINK-14500">FLINK-14500</a>)</strong></p>
+
+<p>From Flink 1.11, you can define and register custom <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/table/python/python_udfs.html#table-functions">UDTFs</a> in PyFlink. Similar to a Python UDF, a UDTF takes zero, one or multiple scalar values as input, but can return an arbitrary number of rows as output instead of a single value.</p>
+
+<p><strong>Cython Performance Optimization for UDFs (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-121%3A+Support+Cython+Optimizing+Python+User+Defined+Function">FLIP-121</a>)</strong></p>
+
+<p><a href="https://cython.readthedocs.io/en/latest/src/quickstart/cythonize.html">Cython</a> is a compiled superset of the Python language that is often used to improve the performance of large-scale numeric processing in Python, as it optimizes execution to machine code-level speed and pairs well with popular C-based libraries like NumPy. From Flink 1.11, you can build <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.11/flinkDev/bui [...]
+
+<p><strong>User-defined Metrics in Python UDFs (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-112%3A+Support+User-Defined+Metrics+in++Python+UDF">FLIP-112</a>)</strong></p>
+
+<p>To make it easier for users to monitor and debug the execution of Python UDFs, PyFlink now allows gathering and exposing metrics to external systems, as well as defining user scopes and variables. You can access the metrics system from a UDF by calling <code>function_context.get_metric_group()</code> in the open method, as described in the <a href="https://ci.apache.org/projects/flink/flink-docs-master/dev/table/python/metrics.html#registering-metrics"&g [...]
+
+<hr />
+
+<h2 id="important-changes">Important Changes</h2>
+
+<ul>
+ <li>
+ <p>[<a href="https://jira.apache.org/jira/browse/FLINK-17339">FLINK-17339</a>] The Blink planner is the <strong>default</strong> in the Table API/SQL starting from Flink 1.11. This was already the case for the SQL Client since Flink 1.10. The old Flink planner is still supported, but not actively developed.</p>
+ </li>
+ <li>
+ <p>[<a href="https://issues.apache.org/jira/browse/FLINK-5763">FLINK-5763</a>] Savepoints now contain all their state inside a single directory (both metadata and program state). This makes it straightforward to figure out which files make up the state of a savepoint and allows users to <strong>relocate savepoints</strong> by simply moving a directory.</p>
+ </li>
+ <li>
+ <p>[<a href="https://issues.apache.org/jira/browse/FLINK-16408">FLINK-16408</a>] To reduce pressure on the JVM metaspace, the user code class loader is being reused by a <code>TaskExecutor</code> as long as there is at least a single slot allocated for the respective job. This changes Flink’s recovery behaviour slightly, so that it will not reload static fields.</p>
+ </li>
+ <li>
+ <p>[<a href="https://issues.apache.org/jira/browse/FLINK-11086">FLINK-11086</a>] Flink now supports Hadoop versions above <strong>Hadoop 3.0.0</strong>. Note that the Flink project does not provide any updated “flink-shaded-hadoop-*” jars. Users need to provide Hadoop dependencies through the <code>HADOOP_CLASSPATH</code> environment variable (recommended) or the lib/ folder.</p>
+ </li>
+ <li>
+ <p>[<a href="https://issues.apache.org/jira/browse/FLINK-16963">FLINK-16963</a>] All <code>MetricReporters</code> that come with Flink have been converted to plugins. These should no longer be placed into <code>/lib</code> (which may result in dependency conflicts), but <code>/plugins/&lt;some_directory&gt;</code> instead.</p>
+ </li>
+ <li>
+ <p>[<a href="https://issues.apache.org/jira/browse/FLINK-12639">FLINK-12639</a>] The Flink <strong>documentation</strong> is undergoing some <strong>rework</strong>, so you might notice that the navigation and organization of content look slightly different starting from Flink 1.11.</p>
+ </li>
+</ul>
+
+<h2 id="release-notes">Release Notes</h2>
+
+<p>Please review the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.11/release-notes/flink-1.11.html">release notes</a> carefully for a detailed list of changes and new features if you plan to upgrade your setup to Flink 1.11. This version is API-compatible with previous 1.x releases for APIs annotated with the @Public annotation.</p>
+
+<h2 id="list-of-contributors">List of Contributors</h2>
+
+<p>The Apache Flink community would like to thank all the 200+ contributors that have made this release possible:</p>
+
+<p>Aitozi, Alexander Fedulov, Alexey Trenikhin, Aljoscha Krettek, Andrey Zagrebin, Arvid Heise, Ayush Saxena, Bairos, Bartosz Krasinski, Benchao Li, Benoit Hanotte, Benoît Paris, Bhagavan Das, Canbin Zheng, Cedric Chen, Chesnay Schepler, Colm O hEigeartaigh, Congxian Qiu, CrazyTomatoOo, Danish Amjad, Danny Chan, David Anderson, Dawid Wysakowicz, Dian Fu, Dominik Wosiński, Echo Lee, Ethan Marsh, Etienne Chauchot, Fabian Hueske, Fabian Paul, Flavio Pompermaier, Gao Yun, Gary Yao, Ghi [...]
+</description>
+<pubDate>Mon, 06 Jul 2020 10:00:00 +0200</pubDate>
+<link>https://flink.apache.org/news/2020/07/06/release-1.11.0.html</link>
+<guid isPermaLink="true">/news/2020/07/06/release-1.11.0.html</guid>
+</item>
+
+<item>
+<title>Flink on Zeppelin Notebooks for Interactive Data Analysis - Part 2</title>
+<description><p>In a previous post, we introduced the basics of Flink on Zeppelin and how to do Streaming ETL. In this second part of the “Flink on Zeppelin” series of posts, I will share how to
+perform streaming data visualization via Flink on Zeppelin and how to use Apache Flink UDFs in Zeppelin.</p>
+
+<h1 id="streaming-data-visualization">Streaming Data Visualization</h1>
+
+<p>With <a href="https://zeppelin.apache.org/">Zeppelin</a>, you can build a real time streaming dashboard without writing any line of javascript/html/css code.</p>
+
+<p>Overall, Zeppelin supports 3 kinds of streaming data analytics:</p>
+
+<ul>
+ <li>Single Mode</li>
+ <li>Update Mode</li>
+ <li>Append Mode</li>
+</ul>
+
+<h3 id="single-mode">Single Mode</h3>
+<p>Single mode is used for cases when the result of a SQL statement is always one row, such as the following example.
+The output format is translated in HTML, and you can specify a paragraph local property template for the final output content template.
+And you can use <code>{i}</code> as placeholder for the {i}th column of the result.</p>
+
+<center>
+<img src="/img/blog/2020-06-23-flink-on-zeppelin-part2/flink_single_mode.gif" width="80%" alt="Single Mode" />
+</center>
+
+<h3 id="update-mode">Update Mode</h3>
+<p>Update mode is suitable for the cases when the output format is more than one row,
+and will always be continuously updated. Here’s one example where we use <code>GROUP BY</code>.</p>
+
+<center>
+<img src="/img/blog/2020-06-23-flink-on-zeppelin-part2/flink_update_mode.gif" width="80%" alt="Update Mode" />
+</center>
+
+<h3 id="append-mode">Append Mode</h3>
+<p>Append mode is suitable for the cases when the output data is always appended.
+For instance, the example below uses a tumble window.</p>
+
+<center>
+<img src="/img/blog/2020-06-23-flink-on-zeppelin-part2/flink_append_mode.gif" width="80%" alt="Append Mode" />
+</center>
+
+<h1 id="udf">UDF</h1>
+
+<p>SQL is a very powerful language, especially in expressing data flow. But most of the time, you need to handle complicated business logic that cannot be expressed by SQL.
+In these cases UDFs (user-defined functions) come particularly handy. In Zeppelin, you can write Scala or Python UDFs, while you can also import Scala, Python and Java UDFs.
+Here are 2 examples of Scala and Python UDFs:</p>
+
+<ul>
+ <li>Scala UDF</li>
+</ul>
+
+<div class="highlight"><pre><code class="language-scala"><span class="o">%</span><span class="n">flink</span>
+
+<span class="k">class</span> <span class="nc">ScalaUpper</span> <span class="k">extends</span> <span class="nc">ScalarFunction</span> <span class="o">{</span>
+<span class="k">def</span> <span class="n">eval</span><span class="o">(</span><span class="n">str</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span> <span class="k">=</span> <span class="n">str</span><span class="o">.</span><span clas [...]
+<span class="o">}</span>
+<span class="n">btenv</span><span class="o">.</span><span class="n">registerFunction</span><span class="o">(</span><span class="s">&quot;scala_upper&quot;</span><span class="o">,</span> <span class="k">new</span> <span class="nc">ScalaUpper</span><span class="o">())</span></code> [...]
+
+<ul>
+ <li>Python UDF</li>
+</ul>
+
+<div class="highlight"><pre><code class="language-python"><span class="o">%</span><span class="n">flink</span><span class="o">.</span><span class="n">pyflink</span>
+
+<span class="k">class</span> <span class="nc">PythonUpper</span><span class="p">(</span><span class="n">ScalarFunction</span><span class="p">):</span>
+<span class="k">def</span> <span class="nf">eval</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">s</span><span class="p">):</span>
+ <span class="k">return</span> <span class="n">s</span><span class="o">.</span><span class="n">upper</span><span class="p">()</span>
+
+<span class="n">bt_env</span><span class="o">.</span><span class="n">register_function</span><span class="p">(</span><span class="s">&quot;python_upper&quot;</span><span class="p">,</span> <span class="n">udf</span><span class="p">(</span><span class="n">PythonUpper</span><span clas [...]
+
+<p>After you define the UDFs, you can use them directly in SQL:</p>
+
+<ul>
+ <li>Use Scala UDF in SQL</li>
+</ul>
+
+<center>
+<img src="/img/blog/2020-06-23-flink-on-zeppelin-part2/flink_scala_udf.png" width="100%" alt="Scala UDF" />
+</center>
+
+<ul>
+ <li>Use Python UDF in SQL</li>
+</ul>
+
+<center>
+<img src="/img/blog/2020-06-23-flink-on-zeppelin-part2/flink_python_udf.png" width="100%" alt="Python UDF" />
+</center>
+
+<h1 id="summary">Summary</h1>
+
+<p>In this post, we explained how to perform streaming data visualization via Flink on Zeppelin and how to use UDFs.
+Besides that, you can do more in Zeppelin with Flink, such as batch processing, Hive integration and more.
+You can check the following articles for more details and here’s a list of <a href="https://www.youtube.com/watch?v=YxPo0Fosjjg&amp;list=PL4oy12nnS7FFtg3KV1iS5vDb0pTz12VcX">Flink on Zeppelin tutorial videos</a> for your reference.</p>
+
+<h1 id="references">References</h1>
+
+<ul>
+ <li><a href="http://zeppelin.apache.org">Apache Zeppelin official website</a></li>
+ <li>Flink on Zeppelin tutorials - <a href="https://medium.com/@zjffdu/flink-on-zeppelin-part-1-get-started-2591aaa6aa47">Part 1</a></li>
+ <li>Flink on Zeppelin tutorials - <a href="https://medium.com/@zjffdu/flink-on-zeppelin-part-2-batch-711731df5ad9">Part 2</a></li>
+ <li>Flink on Zeppelin tutorials - <a href="https://medium.com/@zjffdu/flink-on-zeppelin-part-3-streaming-5fca1e16754">Part 3</a></li>
+ <li>Flink on Zeppelin tutorials - <a href="https://medium.com/@zjffdu/flink-on-zeppelin-part-4-advanced-usage-998b74908cd9">Part 4</a></li>
+ <li><a href="https://www.youtube.com/watch?v=YxPo0Fosjjg&amp;list=PL4oy12nnS7FFtg3KV1iS5vDb0pTz12VcX">Flink on Zeppelin tutorial videos</a></li>
+</ul>
+</description>
+<pubDate>Tue, 23 Jun 2020 10:00:00 +0200</pubDate>
+<link>https://flink.apache.org/ecosystem/2020/06/23/flink-on-zeppelin-part2.html</link>
+<guid isPermaLink="true">/ecosystem/2020/06/23/flink-on-zeppelin-part2.html</guid>
+</item>
+
+<item>
+<title>Flink on Zeppelin Notebooks for Interactive Data Analysis - Part 1</title>
+<description><p>The latest release of <a href="https://zeppelin.apache.org/">Apache Zeppelin</a> comes with a redesigned interpreter for Apache Flink (version Flink 1.10+ is only supported moving forward)
+that allows developers to use Flink directly on Zeppelin notebooks for interactive data analysis. I wrote 2 posts about how to use Flink in Zeppelin. This is part-1 where I explain how the Flink interpreter in Zeppelin works,
+and provide a tutorial for running Streaming ETL with Flink on Zeppelin.</p>
+
+<h1 id="the-flink-interpreter-in-zeppelin-09">The Flink Interpreter in Zeppelin 0.9</h1>
+
+<p>The Flink interpreter can be accessed and configured from Zeppelin’s interpreter settings page.
+The interpreter has been refactored so that Flink users can now take advantage of Zeppelin to write Flink applications in three languages,
+namely Scala, Python (PyFlink) and SQL (for both batch &amp; streaming executions).
+Zeppelin 0.9 now comes with the Flink interpreter group, consisting of the below five interpreters:</p>
+
+<ul>
+ <li>%flink - Provides a Scala environment</li>
+ <li>%flink.pyflink - Provides a python environment</li>
+ <li>%flink.ipyflink - Provides an ipython environment</li>
+ <li>%flink.ssql - Provides a stream sql environment</li>
+ <li>%flink.bsql - Provides a batch sql environment</li>
+</ul>
+
+<p>Not only has the interpreter been extended to support writing Flink applications in three languages, but it has also extended the available execution modes for Flink that now include:</p>
+
+<ul>
+ <li>Running Flink in Local Mode</li>
+ <li>Running Flink in Remote Mode</li>
+ <li>Running Flink in Yarn Mode</li>
+</ul>
+
+<p>You can find more information about how to get started with Zeppelin and all the execution modes for Flink applications in <a href="https://github.com/apache/zeppelin/tree/master/notebook/Flink%20Tutorial">Zeppelin notebooks</a> in this post.</p>
+
+<h1 id="flink-on-zeppelin-for-stream-processing">Flink on Zeppelin for Stream processing</h1>
+
+<p>Performing stream processing jobs with Apache Flink on Zeppelin allows you to run most major streaming cases,
+such as streaming ETL and real time data analytics, with the use of Flink SQL and specific UDFs.
+Below we showcase how you can execute streaming ETL using Flink on Zeppelin:</p>
+
+<p>You can use Flink SQL to perform streaming ETL by following the steps below
+(for the full tutorial, please refer to the <a href="https://github.com/apache/zeppelin/blob/master/notebook/Flink%20Tutorial/4.%20Streaming%20ETL_2EYD56B9B.zpln">Flink Tutorial/Streaming ETL tutorial</a> of the Zeppelin distribution):</p>
+
+<ul>
+ <li>Step 1. Create source table to represent the source data.</li>
+</ul>
+
+<center>
+<img src="/img/blog/2020-06-15-flink-on-zeppelin/create_source.png" width="80%" alt="Create Source Table" />
+</center>
+
+<ul>
+ <li>Step 2. Create a sink table to represent the processed data.</li>
+</ul>
+
+<center>
+<img src="/img/blog/2020-06-15-flink-on-zeppelin/create_sink.png" width="80%" alt="Create Sink Table" />
+</center>
+
+<ul>
+ <li>Step 3. After creating the source and sink table, we can insert them to our statement to trigger the stream processing job as the following:</li>
+</ul>
+
+<center>
+<img src="/img/blog/2020-06-15-flink-on-zeppelin/etl.png" width="80%" alt="ETL" />
+</center>
+
+<ul>
+ <li>Step 4. After initiating the streaming job, you can use another SQL statement to query the sink table to verify the results of your job. Here you can see the top 10 records which will be refreshed every 3 seconds.</li>
+</ul>
+
+<center>
+<img src="/img/blog/2020-06-15-flink-on-zeppelin/preview.png" width="80%" alt="Preview" />
+</center>
+
+<h1 id="summary">Summary</h1>
+
+<p>In this post, we explained how the redesigned Flink interpreter works in Zeppelin 0.9.0 and provided some examples for performing streaming ETL jobs with
+Flink and Zeppelin. In the next post, I will talk about how to do streaming data visualization via Flink on Zeppelin.
+Besides that, you can find an additional <a href="https://medium.com/@zjffdu/flink-on-zeppelin-part-2-batch-711731df5ad9">tutorial for batch processing with Flink on Zeppelin</a> as well as using Flink on Zeppelin for
+more advance operations like resource isolation, job concurrency &amp; parallelism, multiple Hadoop &amp; Hive environments and more on our series of posts on Medium.
+And here’s a list of <a href="https://www.youtube.com/watch?v=YxPo0Fosjjg&amp;list=PL4oy12nnS7FFtg3KV1iS5vDb0pTz12VcX">Flink on Zeppelin tutorial videos</a> for your reference.</p>
+
+<h1 id="references">References</h1>
+
+<ul>
+ <li><a href="http://zeppelin.apache.org">Apache Zeppelin official website</a></li>
+ <li>Flink on Zeppelin tutorials - <a href="https://medium.com/@zjffdu/flink-on-zeppelin-part-1-get-started-2591aaa6aa47">Part 1</a></li>
+ <li>Flink on Zeppelin tutorials - <a href="https://medium.com/@zjffdu/flink-on-zeppelin-part-2-batch-711731df5ad9">Part 2</a></li>
+ <li>Flink on Zeppelin tutorials - <a href="https://medium.com/@zjffdu/flink-on-zeppelin-part-3-streaming-5fca1e16754">Part 3</a></li>
+ <li>Flink on Zeppelin tutorials - <a href="https://medium.com/@zjffdu/flink-on-zeppelin-part-4-advanced-usage-998b74908cd9">Part 4</a></li>
+ <li><a href="https://www.youtube.com/watch?v=YxPo0Fosjjg&amp;list=PL4oy12nnS7FFtg3KV1iS5vDb0pTz12VcX">Flink on Zeppelin tutorial videos</a></li>
+</ul>
+</description>
+<pubDate>Mon, 15 Jun 2020 10:00:00 +0200</pubDate>
+<link>https://flink.apache.org/news/2020/06/15/flink-on-zeppelin-part1.html</link>
+<guid isPermaLink="true">/news/2020/06/15/flink-on-zeppelin-part1.html</guid>
+</item>
+
+<item>
+<title>Flink Community Update - June'20</title>
+<description><p>And suddenly it’s June. The previous month has been calm on the surface, but quite hectic underneath — the final testing phase for Flink 1.11 is moving at full speed, Stateful Functions 2.1 is out in the wild and Flink has made it into Google Season of Docs 2020.</p>
+
+<p>To top it off, a piece of good news: <a href="https://www.flink-forward.org/global-2020">Flink Forward</a> is back on October 19-22 as a free virtual event!</p>
+
+<div class="page-toc">
+<ul id="markdown-toc">
+ <li><a href="#the-past-month-in-flink" id="markdown-toc-the-past-month-in-flink">The Past Month in Flink</a> <ul>
+ <li><a href="#flink-stateful-functions-21-release" id="markdown-toc-flink-stateful-functions-21-release">Flink Stateful Functions 2.1 Release</a></li>
+ <li><a href="#testing-is-on-for-flink-111" id="markdown-toc-testing-is-on-for-flink-111">Testing is ON for Flink 1.11</a></li>
+ <li><a href="#flink-minor-releases" id="markdown-toc-flink-minor-releases">Flink Minor Releases</a> <ul>
+ <li><a href="#flink-1101" id="markdown-toc-flink-1101">Flink 1.10.1</a></li>
+ </ul>
+ </li>
+ <li><a href="#new-committers-and-pmc-members" id="markdown-toc-new-committers-and-pmc-members">New Committers and PMC Members</a> <ul>
+ <li><a href="#new-committers" id="markdown-toc-new-committers">New Committers</a></li>
+ </ul>
+ </li>
+ </ul>
+ </li>
+ <li><a href="#the-bigger-picture" id="markdown-toc-the-bigger-picture">The Bigger Picture</a> <ul>
+ <li><a href="#flink-forward-global-virtual-conference-2020" id="markdown-toc-flink-forward-global-virtual-conference-2020">Flink Forward Global Virtual Conference 2020</a></li>
+ <li><a href="#google-season-of-docs-2020" id="markdown-toc-google-season-of-docs-2020">Google Season of Docs 2020</a></li>
+ </ul>
+ </li>
+</ul>
+
+</div>
+
+<h1 id="the-past-month-in-flink">The Past Month in Flink</h1>
+
+<h2 id="flink-stateful-functions-21-release">Flink Stateful Functions 2.1 Release</h2>
+
+<p>It might seem like <a href="https://flink.apache.org/news/2020/04/07/release-statefun-2.0.0.html">Stateful Functions 2.0 was announced</a> only a handful of weeks ago (and it was!), but the Flink community has just released Stateful Functions 2.1! This release introduces two new features: state expiration for any kind of persisted state and support for UNIX Domain Sockets (UDS) to improve the performance of inter-container communication in co-located deploy [...]
+
+<p>As the community around StateFun grows, the release cycle will follow this pattern of smaller and more frequent releases to incorporate user feedback and allow for faster iteration. If you’d like to get involved, we’re always <a href="https://github.com/apache/flink-statefun#contributing">looking for new contributors</a> — especially around SDKs for other languages (e.g. Go, Rust, Javascript).</p>
+
+<hr />
+
+<h2 id="testing-is-on-for-flink-111">Testing is ON for Flink 1.11</h2>
+
+<p>Things have been pretty quiet in the Flink community, as all efforts shifted to testing the newest features shipping with Flink 1.11. While we wait for a voting Release Candidate (RC) to be out, you can check the progress of testing in <a href="https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=364&amp;projectKey=FLINK">this JIRA burndown board</a> and learn more about some of the <a href="https://flink.apache.org/news/2020/05/07/co [...]
+
+<ul>
+ <li>
+ <p><a href="https://www.youtube.com/watch?v=ssEmeLcL5Uk">Rethinking of fault tolerance in Flink: what lies ahead?</a></p>
+ </li>
+ <li>
+ <p><a href="https://www.youtube.com/watch?v=t7fAN3xNJ3Q">It’s finally here: Python on Flink &amp; Flink on Zeppelin</a></p>
+ </li>
+ <li>
+ <p><a href="https://www.youtube.com/watch?v=KDD8e4GE12w">A deep dive into Flink SQL</a></p>
+ </li>
+ <li>
+ <p><a href="https://www.youtube.com/watch?v=4ce1H9CRyEc">Production-Ready Flink and Hive Integration - what story you can tell now?</a></p>
+ </li>
+</ul>
+
+<p>We encourage the wider community to also get involved in testing once the voting RC is out. Keep an eye on the <a href="https://flink.apache.org/community.html#mailing-lists">@dev mailing list</a> for updates!</p>
+
+<hr />
+
+<h2 id="flink-minor-releases">Flink Minor Releases</h2>
+
+<h3 id="flink-1101">Flink 1.10.1</h3>
+
+<p>The community released Flink 1.10.1, covering some outstanding bugs in Flink 1.10. You can find more in the <a href="https://flink.apache.org/news/2020/05/12/release-1.10.1.html">announcement blogpost</a>!</p>
+
+<hr />
+
+<h2 id="new-committers-and-pmc-members">New Committers and PMC Members</h2>
+
+<p>The Apache Flink community has welcomed <strong>2 new Committers</strong> since the last update. Congratulations!</p>
+
+<h3 id="new-committers">New Committers</h3>
+
+<div class="row">
+ <div class="col-lg-3">
+ <div class="text-center">
+ <img class="img-circle" src="https://avatars3.githubusercontent.com/u/4471524?s=400&amp;v=4" width="90" height="90" />
+ <p>Benchao Li</p>
+ </div>
+ </div>
+ <div class="col-lg-3">
+ <div class="text-center">
+ <img class="img-circle" src="https://avatars0.githubusercontent.com/u/6509172?s=400&amp;v=4" width="90" height="90" />
+ <p>Xintong Song</p>
+ </div>
+ </div>
+</div>
+
+<hr />
+
+<h1 id="the-bigger-picture">The Bigger Picture</h1>
+
+<h2 id="flink-forward-global-virtual-conference-2020">Flink Forward Global Virtual Conference 2020</h2>
+
+<p>After a first successful <a href="https://www.youtube.com/playlist?list=PLDX4T_cnKjD0ngnBSU-bYGfgVv17MiwA7">virtual conference</a> last April, Flink Forward will be hosting a second free virtual edition on October 19-22. This time around, the conference will feature two days of hands-on training and two full days of conference talks!</p>
+
+<p>Got a Flink story to share? Maybe your recent adventures with Stateful Functions? The <a href="https://www.flink-forward.org/global-2020/call-for-presentations">Call for Presentations is now open</a> and accepting submissions from the community until <strong>June 19th, 11:59 PM CEST</strong>.</p>
+
+<div style="line-height:60%;">
+ <br />
+</div>
+
+<center>
+<img src="/img/blog/2020-06-10-community-update/FlinkForward_Banner_CFP_Global_2020.png" width="600px" alt="Flink Forward Global 2020" />
+</center>
+
+<div style="line-height:60%;">
+ <br />
+</div>
+
+<hr />
+
+<h2 id="google-season-of-docs-2020">Google Season of Docs 2020</h2>
+
+<p>In the last update, we announced that Flink was applying to <a href="https://developers.google.com/season-of-docs">Google Season of Docs (GSoD)</a> again this year. The good news: we’ve made it into the shortlist of accepted projects! This represents an invaluable opportunity for the Flink community to collaborate with technical writers to improve the Table API &amp; SQL documentation. We’re honored to have seen a great number of people reach out over t [...]
+
+<p>If you’re interested in learning more about our project idea or want to get involved in GSoD as a technical writer, check out the <a href="https://flink.apache.org/news/2020/05/04/season-of-docs.html">announcement blogpost</a> and <a href="https://developers.google.com/season-of-docs/docs/tech-writer-application-hints">submit your application</a>. The deadline for GSoD applications is <strong>July 9th, 18:00 UTC</strong>.&l [...]
+
+<hr />
+
+<p>If you’d like to keep a closer eye on what’s happening in the community, subscribe to the Flink <a href="https://flink.apache.org/community.html#mailing-lists">@community mailing list</a> to get fine-grained weekly updates, upcoming event announcements and more.</p>
+</description>
+<pubDate>Thu, 11 Jun 2020 10:00:00 +0200</pubDate>
+<link>https://flink.apache.org/news/2020/06/11/community-update.html</link>
+<guid isPermaLink="true">/news/2020/06/11/community-update.html</guid>
+</item>
+
+<item>
+<title>Stateful Functions 2.1.0 Release Announcement</title>
+<description><p>The Apache Flink community is happy to announce the release of Stateful Functions (StateFun) 2.1.0! This release introduces new features around state expiration and performance improvements for co-located deployments, as well as other important changes that improve the stability and testability of the project. As the community around StateFun grows, the release cycle will follow this pattern of smaller and more frequent releases to incorporate user feedback and allo [...]
+
+<p>The binary distribution and source artifacts are now available on the updated <a href="https://flink.apache.org/downloads.html">Downloads</a> page of the Flink website, and the most recent Python SDK distribution is available on <a href="https://pypi.org/project/apache-flink-statefun/">PyPI</a>. For more details, check the complete <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&amp;versio [...]
+
+<div class="page-toc">
+<ul id="markdown-toc">
+ <li><a href="#new-features-and-improvements" id="markdown-toc-new-features-and-improvements">New Features and Improvements</a> <ul>
+ <li><a href="#support-for-state-time-to-live-ttl" id="markdown-toc-support-for-state-time-to-live-ttl">Support for State Time-To-Live (TTL)</a></li>
+ <li><a href="#improved-performance-with-unix-domain-sockets-uds" id="markdown-toc-improved-performance-with-unix-domain-sockets-uds">Improved Performance with UNIX Domain Sockets (UDS)</a></li>
+ </ul>
+ </li>
+ <li><a href="#important-changes" id="markdown-toc-important-changes">Important Changes</a></li>
+ <li><a href="#release-notes" id="markdown-toc-release-notes">Release Notes</a></li>
+ <li><a href="#list-of-contributors" id="markdown-toc-list-of-contributors">List of Contributors</a></li>
+</ul>
+
+</div>
+
+<h2 id="new-features-and-improvements">New Features and Improvements</h2>
+
+<h3 id="support-for-state-time-to-live-ttl">Support for State Time-To-Live (TTL)</h3>
+
+<p>Being able to define state expiration and a state cleanup strategy is a useful feature for stateful applications — for example, to keep state size from growing indefinitely or to work with sensitive data. In previous StateFun versions, users could implement this behavior manually using <a href="https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.0/sdk/java.html#sending-delayed-messages">delayed messages</a> as state expiration callbacks. For [...]
+
+<p>Persisted state can be configured to expire after the last <em>write</em> operation (<code>AFTER_WRITE</code>) or after the last <em>read or write</em> operation (<code>AFTER_READ_AND_WRITE</code>). For the <a href="https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.1/sdk/java.html#state-expiration">Java SDK</a>, users can configure State TTL in the definition of their persisted fields:</p>
+
+<div class="highlight"><pre><code class="language-java"><span class="nd">@Persisted</span>
+<span class="n">PersistedValue</span><span class="o">&lt;</span><span class="n">Integer</span><span class="o">&gt;</span> <span class="n">table</span> <span class="o">=</span> <span class="n">PersistedValue</span><span class="o">.</span><span class="na">of</span><span class="o& [...]
+ <span class="s">&quot;my-value&quot;</span><span class="o">,</span>
+ <span class="n">Integer</span><span class="o">.</span><span class="na">class</span><span class="o">,</span>
+ <span class="n">Expiration</span><span class="o">.</span><span class="na">expireAfterWriting</span><span class="o">(</span><span class="n">Duration</span><span class="o">.</span><span class="na">ofHours</span><span class="o">(</span><span class="mi">1</span><span class="o"> [...]
+
+<p>For <a href="https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.1/concepts/distributed_architecture.html#remote-functions">remote functions</a> using e.g. the Python SDK, users can configure State TTL in their <code>module.yaml</code>:</p>
+
+<div class="highlight"><pre><code>functions:
+ - function:
+ states:
+ - name: xxxx
+ expireAfter: 5min # optional key
+</code></pre></div>
+
+<div class="alert alert-info">
+ <b>Note:</b>
+ The state expiration mode for remote functions is currently restricted to AFTER_READ_AND_WRITE, and the actual TTL being set is the longest duration across all registered state, not for each individual state entry. This is planned to be improved in upcoming releases (<a href="https://issues.apache.org/jira/browse/FLINK-17954">FLINK-17954</a>).
+</div>
+
+<h3 id="improved-performance-with-unix-domain-sockets-uds">Improved Performance with UNIX Domain Sockets (UDS)</h3>
+
+<p>Stateful functions can be <a href="https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.1/concepts/distributed_architecture.html#deployment-styles-for-functions">deployed in multiple ways</a>, even within the same application. For deployments where functions are <a href="https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.1/concepts/distributed_architecture.html#co-located-functions">co-located</a> with the [...]
+
+<p>Users can <a href="https://ci.apache.org/projects/flink/flink-statefun-docs-master/sdk/modules.html#defining-functions">enable transport via UDS</a> in a remote module by specifying the following in their <code>module.yaml</code>:</p>
+
+<div class="highlight"><pre><code>functions:
+ - function:
+ spec:
+ - endpoint: http(s)+unix://&lt;socket-file-path&gt;/&lt;serve-url-path&gt;
+</code></pre></div>
+
+<h2 id="important-changes">Important Changes</h2>
+
+<ul>
+ <li>
+ <p>[<a href="https://issues.apache.org/jira/browse/FLINK-17712">FLINK-17712</a>] The Flink version in StateFun 2.1 has been upgraded to 1.10.1, the most recent patch version.</p>
+ </li>
+ <li>
+ <p>[<a href="https://issues.apache.org/jira/browse/FLINK-17533">FLINK-17533</a>] StateFun 2.1 now supports concurrent checkpoints, which means applications will no longer fail on savepoints that are triggered concurrently to a checkpoint.</p>
+ </li>
+ <li>
+ <p>[<a href="https://issues.apache.org/jira/browse/FLINK-16928">FLINK-16928</a>] StateFun 2.0 was using the Flink legacy scheduler due to a <a href="https://issues.apache.org/jira/browse/FLINK-16927">bug in Flink 1.10</a>. In 2.1, this change is reverted to using the new Flink scheduler again.</p>
+ </li>
+ <li>
+ <p>[<a href="https://issues.apache.org/jira/browse/FLINK-17516">FLINK-17516</a>] The coverage for end-to-end StateFun tests has been extended to also include exactly-once semantics verification (with failure recovery).</p>
+ </li>
+</ul>
+
+<h2 id="release-notes">Release Notes</h2>
+
+<p>Please review the <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&amp;version=12347861">release notes</a> for a detailed list of changes and new features if you plan to upgrade your setup to Stateful Functions 2.1.</p>
+
+<h2 id="list-of-contributors">List of Contributors</h2>
+
+<p>The Apache Flink community would like to thank all contributors that have made this release possible:</p>
+
+<p>abc863377, Authuir, Chesnay Schepler, Congxian Qiu, David Anderson, Dian Fu, Francesco Guardiani, Igal Shilman, Marta Paes Moreira, Patrick Wiener, Rafi Aroch, Seth Wiesman, Stephan Ewen, Tzu-Li (Gordon) Tai</p>
+
+<p>If you’d like to get involved, we’re always <a href="https://github.com/apache/flink-statefun#contributing">looking for new contributors</a> — especially around SDKs for other languages like Go, Rust or Javascript.</p>
+</description>
+<pubDate>Tue, 09 Jun 2020 10:00:00 +0200</pubDate>
+<link>https://flink.apache.org/news/2020/06/09/release-statefun-2.1.0.html</link>
+<guid isPermaLink="true">/news/2020/06/09/release-statefun-2.1.0.html</guid>
+</item>
+
+<item>
+<title>Apache Flink 1.10.1 Released</title>
+<description><p>The Apache Flink community released the first bugfix version of the Apache Flink 1.10 series.</p>
+
+<p>This release includes 158 fixes and minor improvements for Flink 1.10.0. The list below includes a detailed list of all fixes and improvements.</p>
+
+<p>We highly recommend all users to upgrade to Flink 1.10.1.</p>
+
+<div class="alert alert-info">
+ <p><span class="label label-info" style="display: inline-block"><span class="glyphicon glyphicon-info-sign" aria-hidden="true"></span> Note</span>
+FLINK-16684 changed the builders of the StreamingFileSink to make them compilable in Scala. This change is source compatible but binary incompatible. If using the StreamingFileSink, please recompile your user code against 1.10.1 before upgrading.</p>
+</div>
+
+<div class="alert alert-info">
+ <p><span class="label label-info" style="display: inline-block"><span class="glyphicon glyphicon-info-sign" aria-hidden="true"></span> Note</span>
+FLINK-16683 Flink no longer supports starting clusters with .bat scripts. Users should instead use environments like WSL or Cygwin and work with the .sh scripts.</p>
+</div>
+
+<p>Updated Maven dependencies:</p>
+
+<div class="highlight"><pre><code class="language-xml"><span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-java<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.10.1<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span>
+<span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-streaming-java_2.11<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.10.1<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span>
+<span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-clients_2.11<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.10.1<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span></code></pre></div>
+
+<p>You can find the binaries on the updated <a href="/downloads.html">Downloads page</a>.</p>
+
+<p>List of resolved issues:</p>
+
+<h2> Sub-task
+</h2>
+<ul>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14126">FLINK-14126</a>] - Elasticsearch Xpack Machine Learning doesn&#39;t support ARM
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15143">FLINK-15143</a>] - Create document for FLIP-49 TM memory model and configuration guide
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15561">FLINK-15561</a>] - Unify Kerberos credentials checking
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15790">FLINK-15790</a>] - Make FlinkKubeClient and its implementations asynchronous
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15817">FLINK-15817</a>] - Kubernetes Resource leak while deployment exception happens
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16049">FLINK-16049</a>] - Remove outdated &quot;Best Practices&quot; section from Application Development Section
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16131">FLINK-16131</a>] - Translate &quot;Amazon S3&quot; page of &quot;File Systems&quot; into Chinese
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16389">FLINK-16389</a>] - Bump Kafka 0.10 to 0.10.2.2
+</li>
+</ul>
+
+<h2> Bug
+</h2>
+<ul>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-2336">FLINK-2336</a>] - ArrayIndexOufOBoundsException in TypeExtractor when mapping
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10918">FLINK-10918</a>] - incremental Keyed state with RocksDB throws cannot create directory error in windows
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11193">FLINK-11193</a>] - Rocksdb timer service factory configuration option is not settable per job
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13483">FLINK-13483</a>] - PrestoS3FileSystemITCase.testDirectoryListing fails on Travis
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14038">FLINK-14038</a>] - ExecutionGraph deploy failed due to akka timeout
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14311">FLINK-14311</a>] - Streaming File Sink end-to-end test failed on Travis
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14316">FLINK-14316</a>] - Stuck in &quot;Job leader ... lost leadership&quot; error
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15417">FLINK-15417</a>] - Remove the docker volume or mount when starting Mesos e2e cluster
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15669">FLINK-15669</a>] - SQL client can&#39;t cancel flink job
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15772">FLINK-15772</a>] - Shaded Hadoop S3A with credentials provider end-to-end test fails on travis
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15811">FLINK-15811</a>] - StreamSourceOperatorWatermarksTest.testNoMaxWatermarkOnAsyncCancel fails on Travis
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15812">FLINK-15812</a>] - HistoryServer archiving is done in Dispatcher main thread
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15838">FLINK-15838</a>] - Dangling CountDownLatch.await(timeout)
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15852">FLINK-15852</a>] - Job is submitted to the wrong session cluster
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15904">FLINK-15904</a>] - Make Kafka Consumer work with activated &quot;disableGenericTypes()&quot;
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15936">FLINK-15936</a>] - TaskExecutorTest#testSlotAcceptance deadlocks
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15953">FLINK-15953</a>] - Job Status is hard to read for some Statuses
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16013">FLINK-16013</a>] - List and map config options could not be parsed correctly
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16014">FLINK-16014</a>] - S3 plugin ClassNotFoundException SAXParser
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16025">FLINK-16025</a>] - Service could expose blob server port mismatched with JM Container
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16026">FLINK-16026</a>] - Travis failed due to python setup
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16047">FLINK-16047</a>] - Blink planner produces wrong aggregate results with state clean up
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16067">FLINK-16067</a>] - Flink&#39;s CalciteParser swallows error position information
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16068">FLINK-16068</a>] - table with keyword-escaped columns and computed_column_expression columns
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16070">FLINK-16070</a>] - Blink planner can not extract correct unique key for UpsertStreamTableSink
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16108">FLINK-16108</a>] - StreamSQLExample is failed if running in blink planner
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16111">FLINK-16111</a>] - Kubernetes deployment does not respect &quot;taskmanager.cpu.cores&quot;.
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16113">FLINK-16113</a>] - ExpressionReducer shouldn&#39;t escape the reduced string value
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16115">FLINK-16115</a>] - Aliyun oss filesystem could not work with plugin mechanism
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16139">FLINK-16139</a>] - Co-location constraints are not reset on task recovery in DefaultScheduler
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16161">FLINK-16161</a>] - Statistics zero should be unknown in HiveCatalog
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16170">FLINK-16170</a>] - SearchTemplateRequest ClassNotFoundException when use flink-sql-connector-elasticsearch7
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16220">FLINK-16220</a>] - JsonRowSerializationSchema throws cast exception : NullNode cannot be cast to ArrayNode
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16231">FLINK-16231</a>] - Hive connector is missing jdk.tools exclusion against Hive 2.x.x
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16234">FLINK-16234</a>] - Fix unstable cases in StreamingJobGraphGeneratorTest
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16241">FLINK-16241</a>] - Remove the license and notice file in flink-ml-lib module on release-1.10 branch
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16242">FLINK-16242</a>] - BinaryGeneric serialization error cause checkpoint failure
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16262">FLINK-16262</a>] - Class loader problem with FlinkKafkaProducer.Semantic.EXACTLY_ONCE and usrlib directory
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16269">FLINK-16269</a>] - Generic type can not be matched when convert table to stream.
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16281">FLINK-16281</a>] - parameter &#39;maxRetryTimes&#39; can not work in JDBCUpsertTableSink
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16301">FLINK-16301</a>] - Annoying &quot;Cannot find FunctionDefinition&quot; messages with SQL for f_proctime or =
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16308">FLINK-16308</a>] - SQL connector download links are broken
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16313">FLINK-16313</a>] - flink-state-processor-api: surefire execution unstable on Azure
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16331">FLINK-16331</a>] - Remove source licenses for old WebUI
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16345">FLINK-16345</a>] - Computed column can not refer time attribute column
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16360">FLINK-16360</a>] - connector on hive 2.0.1 don&#39;t support type conversion from STRING to VARCHAR
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16371">FLINK-16371</a>] - HadoopCompressionBulkWriter fails with &#39;java.io.NotSerializableException&#39;
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16373">FLINK-16373</a>] - EmbeddedLeaderService: IllegalStateException: The RPC connection is already closed
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16413">FLINK-16413</a>] - Reduce hive source parallelism when limit push down
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16414">FLINK-16414</a>] - create udaf/udtf function using sql casuing ValidationException: SQL validation failed. null
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16433">FLINK-16433</a>] - TableEnvironmentImpl doesn&#39;t clear buffered operations when it fails to translate the operation
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16435">FLINK-16435</a>] - Replace since decorator with versionadd to mark the version an API was introduced
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16467">FLINK-16467</a>] - MemorySizeTest#testToHumanReadableString() is not portable
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16526">FLINK-16526</a>] - Fix exception when computed column expression references a keyword column name
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16541">FLINK-16541</a>] - Document of table.exec.shuffle-mode is incorrect
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16550">FLINK-16550</a>] - HadoopS3* tests fail with NullPointerException exceptions
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16560">FLINK-16560</a>] - Forward Configuration in PackagedProgramUtils#getPipelineFromProgram
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16567">FLINK-16567</a>] - Get the API error of the StreamQueryConfig on Page &quot;Query Configuration&quot;
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16573">FLINK-16573</a>] - Kinesis consumer does not properly shutdown RecordFetcher threads
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16576">FLINK-16576</a>] - State inconsistency on restore with memory state backends
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16626">FLINK-16626</a>] - Prevent REST handler from being closed more than once
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16632">FLINK-16632</a>] - SqlDateTimeUtils#toSqlTimestamp(String, String) may yield incorrect result
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16635">FLINK-16635</a>] - Incompatible okio dependency in flink-metrics-influxdb module
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16646">FLINK-16646</a>] - flink read orc file throw a NullPointerException
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16647">FLINK-16647</a>] - Miss file extension when inserting to hive table with compression
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16652">FLINK-16652</a>] - BytesColumnVector should init buffer in Hive 3.x
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16662">FLINK-16662</a>] - Blink Planner failed to generate JobGraph for POJO DataStream converting to Table (Cannot determine simple type name)
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16664">FLINK-16664</a>] - Unable to set DataStreamSource parallelism to default (-1)
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16675">FLINK-16675</a>] - TableEnvironmentITCase. testClearOperation fails on travis nightly build
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16684">FLINK-16684</a>] - StreamingFileSink builder does not work with Scala
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16696">FLINK-16696</a>] - Savepoint trigger documentation is insufficient
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16703">FLINK-16703</a>] - AkkaRpcActor state machine does not record transition to terminating state.
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16705">FLINK-16705</a>] - LocalExecutor tears down MiniCluster before client can retrieve JobResult
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16718">FLINK-16718</a>] - KvStateServerHandlerTest leaks Netty ByteBufs
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16727">FLINK-16727</a>] - Fix cast exception when having time point literal as parameters
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16732">FLINK-16732</a>] - Failed to call Hive UDF with constant return value
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16740">FLINK-16740</a>] - OrcSplitReaderUtil::logicalTypeToOrcType fails to create decimal type with precision &lt; 10
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16759">FLINK-16759</a>] - HiveModuleTest failed to compile on release-1.10
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16767">FLINK-16767</a>] - Failed to read Hive table with RegexSerDe
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16771">FLINK-16771</a>] - NPE when filtering by decimal column
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16821">FLINK-16821</a>] - Run Kubernetes test failed with invalid named &quot;minikube&quot;
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16822">FLINK-16822</a>] - The config set by SET command does not work
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16825">FLINK-16825</a>] - PrometheusReporterEndToEndITCase should rely on path returned by DownloadCache
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16836">FLINK-16836</a>] - Losing leadership does not clear rpc connection in JobManagerLeaderListener
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16860">FLINK-16860</a>] - Failed to push filter into OrcTableSource when upgrading to 1.9.2
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16888">FLINK-16888</a>] - Re-add jquery license file under &quot;/licenses&quot;
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16901">FLINK-16901</a>] - Flink Kinesis connector NOTICE should have contents of AWS KPL&#39;s THIRD_PARTY_NOTICES file manually merged in
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16913">FLINK-16913</a>] - ReadableConfigToConfigurationAdapter#getEnum throws UnsupportedOperationException
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16916">FLINK-16916</a>] - The logic of NullableSerializer#copy is wrong
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16944">FLINK-16944</a>] - Compile error in. DumpCompiledPlanTest and PreviewPlanDumpTest
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16980">FLINK-16980</a>] - Python UDF doesn&#39;t work with protobuf 3.6.1
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16981">FLINK-16981</a>] - flink-runtime tests are crashing the JVM on Java11 because of PowerMock
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17062">FLINK-17062</a>] - Fix the conversion from Java row type to Python row type
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17066">FLINK-17066</a>] - Update pyarrow version bounds less than 0.14.0
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17093">FLINK-17093</a>] - Python UDF doesn&#39;t work when the input column is from composite field
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17107">FLINK-17107</a>] - CheckpointCoordinatorConfiguration#isExactlyOnce() is inconsistent with StreamConfig#getCheckpointMode()
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17114">FLINK-17114</a>] - When the pyflink job runs in local mode and the command &quot;python&quot; points to Python 2.7, the startup of the Python UDF worker will fail.
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17124">FLINK-17124</a>] - The PyFlink Job runs into infinite loop if the Python UDF imports job code
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17152">FLINK-17152</a>] - FunctionDefinitionUtil generate wrong resultType and acc type of AggregateFunctionDefinition
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17308">FLINK-17308</a>] - ExecutionGraphCache cachedExecutionGraphs not cleanup cause OOM Bug
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17313">FLINK-17313</a>] - Validation error when insert decimal/varchar with precision into sink using TypeInformation of row
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17334">FLINK-17334</a>] - Flink does not support HIVE UDFs with primitive return types
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17338">FLINK-17338</a>] - LocalExecutorITCase.testBatchQueryCancel test timeout
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17359">FLINK-17359</a>] - Entropy key is not resolved if flink-s3-fs-hadoop is added as a plugin
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17403">FLINK-17403</a>] - Fix invalid classpath in BashJavaUtilsITCase
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17471">FLINK-17471</a>] - Move LICENSE and NOTICE files to root directory of python distribution
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17483">FLINK-17483</a>] - Update flink-sql-connector-elasticsearch7 NOTICE file to correctly reflect bundled dependencies
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17496">FLINK-17496</a>] - Performance regression with amazon-kinesis-producer 0.13.1 in Flink 1.10.x
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17499">FLINK-17499</a>] - LazyTimerService used to register timers via State Processing API incorrectly mixes event time timers with processing time timers
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17514">FLINK-17514</a>] - TaskCancelerWatchdog does not kill TaskManager
+</li>
+</ul>
+
+<h2> New Feature
+</h2>
+<ul>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17275">FLINK-17275</a>] - Add core training exercises
+</li>
+</ul>
+
+<h2> Improvement
+</h2>
+<ul>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-9656">FLINK-9656</a>] - Environment java opts for flink run
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15094">FLINK-15094</a>] - Warning about using private constructor of java.nio.DirectByteBuffer in Java 11
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15584">FLINK-15584</a>] - Give nested data type of ROWs in ValidationException
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15616">FLINK-15616</a>] - Move boot error messages from python-udf-boot.log to taskmanager&#39;s log file
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15989">FLINK-15989</a>] - Rewrap OutOfMemoryError in allocateUnpooledOffHeap with better message
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16018">FLINK-16018</a>] - Improve error reporting when submitting batch job (instead of AskTimeoutException)
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16125">FLINK-16125</a>] - Make zookeeper.connect optional for Kafka connectors
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16167">FLINK-16167</a>] - Update documentation about python shell execution
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16191">FLINK-16191</a>] - Improve error message on Windows when RocksDB Paths are too long
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16280">FLINK-16280</a>] - Fix sample code errors in the documentation about elasticsearch connector
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16288">FLINK-16288</a>] - Setting the TTL for discarding task pods on Kubernetes.
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16293">FLINK-16293</a>] - Document using plugins in Kubernetes
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16343">FLINK-16343</a>] - Improve exception message when reading an unbounded source in batch mode
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16406">FLINK-16406</a>] - Increase default value for JVM Metaspace to minimise its OutOfMemoryError
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16538">FLINK-16538</a>] - Restructure Python Table API documentation
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16604">FLINK-16604</a>] - Column key in JM configuration is too narrow
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16683">FLINK-16683</a>] - Remove scripts for starting Flink on Windows
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16697">FLINK-16697</a>] - Disable JMX rebinding
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16763">FLINK-16763</a>] - Should not use BatchTableEnvironment for Python UDF in the document of flink-1.10
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16772">FLINK-16772</a>] - Bump derby to 10.12.1.1+ or exclude it
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16790">FLINK-16790</a>] - enables the interpretation of backslash escapes
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16862">FLINK-16862</a>] - Remove example url in quickstarts
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16874">FLINK-16874</a>] - Respect the dynamic options when calculating memory options in taskmanager.sh
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16942">FLINK-16942</a>] - ES 5 sink should allow users to select netty transport client
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17065">FLINK-17065</a>] - Add documentation about the Python versions supported for PyFlink
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17125">FLINK-17125</a>] - Add a Usage Notes Page to Answer Common Questions Encountered by PyFlink Users
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17254">FLINK-17254</a>] - Improve the PyFlink documentation and examples to use SQL DDL for source/sink definition
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17276">FLINK-17276</a>] - Add checkstyle to training exercises
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17277">FLINK-17277</a>] - Apply IntelliJ recommendations to training exercises
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17278">FLINK-17278</a>] - Add Travis to the training exercises
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17279">FLINK-17279</a>] - Use gradle build scans for training exercises
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17316">FLINK-17316</a>] - Have HourlyTips solutions use TumblingEventTimeWindows.of
+</li>
+</ul>
+
+<h2> Task
+</h2>
+<ul>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15741">FLINK-15741</a>] - Fix TTL docs after enabling RocksDB compaction filter by default (needs Chinese translation)
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15933">FLINK-15933</a>] - update content of how generic table schema is stored in hive via HiveCatalog
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15991">FLINK-15991</a>] - Create Chinese documentation for FLIP-49 TM memory model
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16004">FLINK-16004</a>] - Exclude flink-rocksdb-state-memory-control-test jars from the dist
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16454">FLINK-16454</a>] - Update the copyright year in NOTICE files
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16530">FLINK-16530</a>] - Add documentation about &quot;GROUPING SETS&quot; and &quot;CUBE&quot; support in streaming mode
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16592">FLINK-16592</a>] - The doc of Streaming File Sink has a mistake of grammar
+</li>
+</ul>
+
+</description>
+<pubDate>Tue, 12 May 2020 14:00:00 +0200</pubDate>
+<link>https://flink.apache.org/news/2020/05/12/release-1.10.1.html</link>
+<guid isPermaLink="true">/news/2020/05/12/release-1.10.1.html</guid>
+</item>
+
+<item>
+<title>Flink Community Update - May'20</title>
+<description><p>Can you smell it? It’s release month! It took a while, but now that we’re <a href="https://flink.apache.org/news/2020/04/01/community-update.html">all caught up with the past</a>, the Community Update is here to stay. This time around, we’re warming up for Flink 1.11 and peeping back to the month of April in the Flink community — with the release of Stateful Functions 2.0, a new self-paced Flink training and some efforts to improve the Flink do [...]
+
+<p>Last month also marked the debut of Flink Forward Virtual Conference 2020: what did you think? If you missed it altogether or just want to recap some of the sessions, the <a href="https://www.youtube.com/playlist?list=PLDX4T_cnKjD0ngnBSU-bYGfgVv17MiwA7">videos</a> and <a href="https://www.slideshare.net/FlinkForward">slides</a> are now available!</p>
+
+<div class="page-toc">
+<ul id="markdown-toc">
+ <li><a href="#the-past-month-in-flink" id="markdown-toc-the-past-month-in-flink">The Past Month in Flink</a> <ul>
+ <li><a href="#flink-stateful-functions-20-is-out" id="markdown-toc-flink-stateful-functions-20-is-out">Flink Stateful Functions 2.0 is out!</a></li>
+ <li><a href="#warming-up-for-flink-111" id="markdown-toc-warming-up-for-flink-111">Warming up for Flink 1.11</a></li>
+ <li><a href="#flink-minor-releases" id="markdown-toc-flink-minor-releases">Flink Minor Releases</a> <ul>
+ <li><a href="#flink-193" id="markdown-toc-flink-193">Flink 1.9.3</a></li>
+ <li><a href="#flink-1101" id="markdown-toc-flink-1101">Flink 1.10.1</a></li>
+ </ul>
+ </li>
+ <li><a href="#new-committers-and-pmc-members" id="markdown-toc-new-committers-and-pmc-members">New Committers and PMC Members</a> <ul>
+ <li><a href="#new-pmc-members" id="markdown-toc-new-pmc-members">New PMC Members</a></li>
+ <li><a href="#new-committers" id="markdown-toc-new-committers">New Committers</a></li>
+ </ul>
+ </li>
+ </ul>
+ </li>
+ <li><a href="#the-bigger-picture" id="markdown-toc-the-bigger-picture">The Bigger Picture</a> <ul>
+ <li><a href="#a-new-self-paced-apache-flink-training" id="markdown-toc-a-new-self-paced-apache-flink-training">A new self-paced Apache Flink training</a></li>
+ <li><a href="#google-season-of-docs-2020" id="markdown-toc-google-season-of-docs-2020">Google Season of Docs 2020</a></li>
+ </ul>
+ </li>
+ <li><a href="#and-something-to-read" id="markdown-toc-and-something-to-read">…and something to read!</a></li>
+</ul>
+
+</div>
+
+<h1 id="the-past-month-in-flink">The Past Month in Flink</h1>
+
+<h2 id="flink-stateful-functions-20-is-out">Flink Stateful Functions 2.0 is out!</h2>
+
+<p>In the beginning of April, the Flink community announced the <a href="https://flink.apache.org/news/2020/04/07/release-statefun-2.0.0.html">release of Stateful Functions 2.0</a> — the first as part of the Apache Flink project. From this release, you can use Flink as the base of a (stateful) serverless platform with out-of-the-box consistent and scalable state, and efficient messaging between functions. You can even run your stateful functions on platforms l [...]
+
+<div style="line-height:60%;">
+ <br />
+</div>
+
+<center>
+<img src="/img/blog/2020-05-06-community-update/2020-05-06-community-update_2.png" width="550px" alt="Stateful Functions" />
+</center>
+
+<div style="line-height:60%;">
+ <br />
+</div>
+
+<p>It’s been encouraging to see so many questions about Stateful Functions popping up in the <a href="https://lists.apache.org/list.html?user@flink.apache.org:lte=3M:statefun">mailing list</a> and Stack Overflow! If you’d like to get involved, we’re always <a href="https://github.com/apache/flink-statefun#contributing">looking for new contributors</a> — especially around SDKs for other languages like Go, Javascript and Rust.</p>
+
+<hr />
+
+<h2 id="warming-up-for-flink-111">Warming up for Flink 1.11</h2>
+
+<p>The final preparations for the release of Flink 1.11 are well underway, with the feature freeze scheduled for May 15th, and there’s a lot of new features and improvements to look out for:</p>
+
+<ul>
+ <li>
+ <p>On the <strong>usability</strong> side, you can expect a big focus on smoothing data ingestion with contributions like support for Change Data Capture (CDC) in the Table API/SQL (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-105%3A+Support+to+Interpret+and+Emit+Changelog+in+Flink+SQL">FLIP-105</a>), easy streaming data ingestion into Apache Hive (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-115%3A [...]
+ </li>
+ <li>
+ <p>On the <strong>operational</strong> side, the much anticipated new Source API (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface">FLIP-27</a>) will unify batch and streaming sources, and improve out-of-the-box event-time behavior; while unaligned checkpoints (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-76%3A+Unaligned+Checkpoints">FLIP-76</a>) and changes [...]
+ </li>
+</ul>
+
+<p>Throw into the mix improvements around type systems, the WebUI, metrics reporting, supported formats and…we can’t wait! To get an overview of the ongoing developments, have a look at <a href="http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/ANNOUNCE-Development-progress-of-Apache-Flink-1-11-tp40718.html">this thread</a>. We encourage the community to get involved in testing once an RC (Release Candidate) is out. Keep an eye on the <a href=& [...]
+
+<hr />
+
+<h2 id="flink-minor-releases">Flink Minor Releases</h2>
+
+<h3 id="flink-193">Flink 1.9.3</h3>
+
+<p>The community released Flink 1.9.3, covering some outstanding bugs from Flink 1.9! You can find more in the <a href="(https://flink.apache.org/news/2020/04/24/release-1.9.3.html)">announcement blogpost</a>.</p>
+
+<h3 id="flink-1101">Flink 1.10.1</h3>
+
+<p>Also in the pipeline is the release of Flink 1.10.1, already in the <a href="http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-Release-1-10-1-release-candidate-2-td41019.html">RC voting</a> phase. So, you can expect Flink 1.10.1 to be released soon!</p>
+
+<hr />
+
+<h2 id="new-committers-and-pmc-members">New Committers and PMC Members</h2>
+
+<p>The Apache Flink community has welcomed <strong>3 PMC Members</strong> and <strong>2 new Committers</strong> since the last update. Congratulations!</p>
+
+<h3 id="new-pmc-members">New PMC Members</h3>
+
+<div class="row">
+ <div class="col-lg-3">
+ <div class="text-center">
+ <img class="img-circle" src="https://avatars2.githubusercontent.com/u/6242259?s=400&amp;u=6e39f4fdbabc8ce4ccde9125166f791957d3ae80&amp;v=4" width="90" height="90" />
+ <p><a href="https://twitter.com/dwysakowicz">Dawid Wysakowicz</a></p>
+ </div>
+ </div>
+ <div class="col-lg-3">
+ <div class="text-center">
+ <img class="img-circle" src="https://avatars1.githubusercontent.com/u/4971479?s=400&amp;u=49d4f217e26186606ab13a17a23a038b62b86682&amp;v=4" width="90" height="90" />
+ <p><a href="https://twitter.com/HequnC">Hequn Cheng</a></p>
+ </div>
+ </div>
+ <div class="col-lg-3">
+ <div class="text-center">
+ <img class="img-circle" src="https://avatars3.githubusercontent.com/u/12387855?s=400&amp;u=37edbfccb6908541f359433f420f9f1bc25bc714&amp;v=4" width="90" height="90" />
+ <p>Zhijiang Wang</p>
+ </div>
+ </div>
+</div>
+
+<h3 id="new-committers">New Committers</h3>
+
+<div class="row">
+ <div class="col-lg-3">
+ <div class="text-center">
+ <img class="img-circle" src="https://avatars3.githubusercontent.com/u/11538663?s=400&amp;u=f4643f1981e2a8f8a1962c34511b0d32a31d9502&amp;v=4" width="90" height="90" />
+ <p><a href="https://twitter.com/snntrable">Konstantin Knauf</a></p>
+ </div>
+ </div>
+ <div class="col-lg-3">
+ <div class="text-center">
+ <img class="img-circle" src="https://avatars1.githubusercontent.com/u/1891970?s=400&amp;u=b7718355ceb1f4a8d1e554c3ae7221e2f32cc8e0&amp;v=4" width="90" height="90" />
+ <p><a href="https://twitter.com/sjwiesman">Seth Wiesman</a></p>
+ </div>
+ </div>
+</div>
+
+<hr />
+
+<h1 id="the-bigger-picture">The Bigger Picture</h1>
+
+<h2 id="a-new-self-paced-apache-flink-training">A new self-paced Apache Flink training</h2>
+
+<div style="line-height:60%;">
+ <br />
+</div>
+
+<p>This week, the Flink website received the invaluable contribution of a self-paced training course curated by David (<a href="https://twitter.com/alpinegizmo">@alpinegizmo</a>) — or, what used to be the entire training materials under <a href="training.ververica.com">training.ververica.com</a>. The new materials guide you through the very basics of Flink and the DataStream API, and round off every concepts section with hands-on exercise [...]
+
+<div style="line-height:60%;">
+ <br />
+</div>
+
+<center>
+<img src="/img/blog/2020-05-06-community-update/2020-05-06-community-update_1.png" width="1000px" alt="Self-paced Flink Training" />
+</center>
+
+<div style="line-height:140%;">
+ <br />
+</div>
+
+<p>Whether you’re new to Flink or just looking to strengthen your foundations, this training is the most comprehensive way to get started and is now completely open source: <a href="https://flink.apache.org/training.html">https://flink.apache.org/training.html</a>. For now, the materials are only available in English, but the community intends to also provide a Chinese translation in the future.</p>
+
+<hr />
+
+<h2 id="google-season-of-docs-2020">Google Season of Docs 2020</h2>
+
+<p>Google Season of Docs (GSOD) is a great initiative organized by <a href="https://opensource.google.com/">Google Open Source</a> to pair technical writers with mentors to work on documentation for open source projects. Last year, the Flink community submitted <a href="https://flink.apache.org/news/2019/04/17/sod.html">an application</a> that unfortunately didn’t make the cut — but we are trying again! This time, with a project idea to i [...]
+
+<p><strong>1) Restructure the Table API &amp; SQL Documentation</strong></p>
+
+<p>Reworking the current documentation structure would allow to:</p>
+
+<ul>
+ <li>
+ <p>Lower the entry barrier to Flink for non-programmatic (i.e. SQL) users.</p>
+ </li>
+ <li>
+ <p>Make the available features more easily discoverable.</p>
+ </li>
+ <li>
+ <p>Improve the flow and logical correlation of topics.</p>
+ </li>
+</ul>
+
+<p><a href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=127405685">FLIP-60</a> contains a detailed proposal on how to reorganize the existing documentation, which can be used as a starting point.</p>
+
+<p><strong>2) Extend the Table API &amp; SQL Documentation</strong></p>
+
+<p>Some areas of the documentation have insufficient detail or are not <a href="https://flink.apache.org/contributing/docs-style.html#general-guiding-principles">accessible</a> for new Flink users. Examples of topics and sections that require attention are: planners, built-in functions, connectors, overview and concepts sections. There is a lot of work to be done and the technical writer could choose what areas to focus on — these improvements could then be ad [...]
+
+<p>If you’re interested in learning more about this project idea or want to get involved in GSoD as a technical writer, check out the <a href="https://flink.apache.org/news/2020/05/04/season-of-docs.html">announcement blogpost</a>.</p>
+
+<hr />
+
+<h1 id="and-something-to-read">…and something to read!</h1>
+
+<p>Events across the globe have pretty much come to a halt, so we’ll leave you with some interesting resources to read and explore instead. In addition to this written content, you can also recap the sessions from the <a href="https://www.youtube.com/playlist?list=PLDX4T_cnKjD0ngnBSU-bYGfgVv17MiwA7">Flink Forward Virtual Conference</a>!</p>
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th>Type</th>
+ <th>Links</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><span class="glyphicon glyphicon glyphicon-bookmark" aria-hidden="true"></span> Blogposts</td>
+ <td><ul>
+ <li><a href="https://medium.com/@abdelkrim.hadjidj/event-driven-supply-chain-for-crisis-with-flinksql-be80cb3ad4f9">Event-Driven Supply Chain for Crisis with FlinkSQL and Zeppelin</a></li>
+ </ul>
+ <ul>
+ <li><a href="https://flink.apache.org/news/2020/04/21/memory-management-improvements-flink-1.10.html">Memory Management Improvements with Apache Flink 1.10</a></li>
+ <li><a href="https://flink.apache.org/news/2020/04/15/flink-serialization-tuning-vol-1.html">Flink Serialization Tuning Vol. 1: Choosing your Serializer — if you can</a></li>
+ </ul>
+ </td>
+ </tr>
+ <tr>
+ <td><span class="glyphicon glyphicon-console" aria-hidden="true"></span> Tutorials</td>
+ <td><ul>
+ <li><a href="https://flink.apache.org/2020/04/09/pyflink-udf-support-flink.html">PyFlink: Introducing Python Support for UDFs in Flink's Table API</a></li>
+ <li><a href="https://dev.to/morsapaes/flink-stateful-functions-where-to-start-2j39">Flink Stateful Functions: where to start?</a></li>
+ </ul>
+ </td>
+ </tr>
+ <tr>
+ <td><span class="glyphicon glyphicon glyphicon-certificate" aria-hidden="true"></span> Flink Packages</td>
+ <td><ul><p><a href="https://flink-packages.org/">Flink Packages</a> is a website where you can explore (and contribute to) the Flink <br /> ecosystem of connectors, extensions, APIs, tools and integrations. <b>New in:</b> </p>
+ <li><a href="https://flink-packages.org/packages/spillable-state-backend-for-flink">Spillable State Backend for Flink</a></li>
+ <li><a href="https://flink-packages.org/packages/flink-memory-calculator">Flink Memory Calculator</a></li>
+ <li><a href="https://flink-packages.org/packages/ververica-platform-community-edition">Ververica Platform Community Edition</a></li>
+ </ul>
+ </td>
+ </tr>
+ </tbody>
+</table>
+
+<p>If you’d like to keep a closer eye on what’s happening in the community, subscribe to the Flink <a href="https://flink.apache.org/community.html#mailing-lists">@community mailing list</a> to get fine-grained weekly updates, upcoming event announcements and more.</p>
+</description>
+<pubDate>Thu, 07 May 2020 10:00:00 +0200</pubDate>
+<link>https://flink.apache.org/news/2020/05/07/community-update.html</link>
+<guid isPermaLink="true">/news/2020/05/07/community-update.html</guid>
+</item>
+
+<item>
+<title>Applying to Google Season of Docs 2020</title>
+<description><p>The Flink community is thrilled to share that the project is applying again to <a href="https://developers.google.com/season-of-docs/">Google Season of Docs</a> (GSoD) this year! If you’re unfamiliar with the program, GSoD is a great initiative organized by <a href="https://opensource.google.com/">Google Open Source</a> to pair technical writers with mentors to work on documentation for open source projects. The <a href [...]
+
+<h1 id="why-apply">Why Apply?</h1>
+
+<p>As one of the most active projects in the ASF, Flink is experiencing a boom in contributions and some major changes to its codebase. And, while the project has also seen a significant increase in activity when it comes to writing, reviewing and translating documentation, it’s hard to keep up with the pace.</p>
+
+<center>
+ <img src="/img/blog/2020-05-04-season-of-docs/2020-04-30-season-of-docs_1.png" width="650px" alt="GitHub 1" />
+</center>
+
+<div style="line-height:60%;">
+ <br />
+</div>
+
+<p>Since last year, the community has been working on <a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-42%3A+Rework+Flink+Documentation">FLIP-42</a> to improve the documentation experience and bring a more accessible information architecture to Flink. After <a href="https://www.mail-archive.com/dev@flink.apache.org/msg36987.html">some discussion</a>, we agreed that GSoD would be a valuable opportunity to double down on this [...]
+
+<h1 id="how-can-you-contribute">How can you contribute?</h1>
+
+<p>If working shoulder to shoulder with the Flink community on documentation sounds exciting, we’d love to hear from you! You can read more about our idea for this year’s project below and, depending on whether it is accepted, <a href="https://developers.google.com/season-of-docs/docs/tech-writer-guide">apply</a> as a technical writer. If you have any questions or just want to know more about the project idea, ping us at <a href="https://flink.apache.o [...]
+
+<div class="alert alert-info">
+ Please <a href="mailto:dev-subscribe@flink.apache.org">subscribe</a> to the Apache Flink mailing list before reaching out.
+ If you are not subscribed then responses to your message will not go through.
+ You can always <a href="mailto:dev-unsubscribe@flink.apache.org">unsubscribe</a> at any time.
+</div>
+
+<h2 id="project-improve-the-table-api--sql-documentation">Project: Improve the Table API &amp; SQL Documentation</h2>
+
+<p><a href="https://flink.apache.org/">Apache Flink</a> is a stateful stream processor supporting a broad set of use cases and featuring APIs at different levels of abstraction that allow users to trade off expressiveness and usability, as well as work with their language of choice (Java/Scala, SQL or Python). The Table API &amp; SQL are Flink’s high-level relational abstractions and focus on data analytics use cases. A core principle is that either API ca [...]
+
+<p>As the Flink community works on extending the scope of the Table API &amp; SQL, a lot of new features are being added and some underlying structures are also being refactored. At the same time, the documentation for these APIs is growing onto a somewhat unruly structure and has potential for improvement in some areas.</p>
+
+<p>The project has two main workstreams: restructuring and extending the Table API &amp; SQL documentation. These can be worked on by one person as a bigger effort or assigned to different technical writers.</p>
+
+<p><strong>1) Restructure the Table API &amp; SQL Documentation</strong></p>
+
+<p>Reworking the current documentation structure would allow to:</p>
+
+<ul>
+ <li>Lower the entry barrier to Flink for non-programmatic (i.e. SQL) users.</li>
+ <li>Make the available features more easily discoverable.</li>
+ <li>Improve the flow and logical correlation of topics.</li>
+</ul>
+
+<p><a href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=127405685">FLIP-60</a> contains a detailed proposal on how to reorganize the existing documentation, which can be used as a starting point.</p>
+
+<p><strong>2) Extend the Table API &amp; SQL Documentation</strong></p>
+
+<p>Some areas of the documentation have insufficient detail or are not <a href="https://flink.apache.org/contributing/docs-style.html#general-guiding-principles">accessible</a> for new Flink users. Examples of topics and sections that require attention are: planners, built-in functions, connectors, overview and concepts sections. There is a lot of work to be done and the technical writer could choose what areas to focus on — these improvements could then be ad [...]
+
+<h3 id="project-mentors">Project Mentors</h3>
+
+<ul>
+ <li><a href="https://twitter.com/aljoscha">Aljoscha Krettek</a> (Apache Flink and Apache Beam PMC Member)</li>
+ <li><a href="https://twitter.com/sjwiesman">Seth Wiesman</a> (Apache Flink Committer)</li>
+</ul>
+
+<h3 id="related-resources">Related Resources</h3>
+
+<ul>
+ <li>
+ <p>FLIP-60: <a href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=127405685">https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=127405685</a></p>
+ </li>
+ <li>
+ <p>Table API &amp; SQL Documentation: <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/">https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/</a></p>
+ </li>
+ <li>
+ <p>How to Contribute Documentation: <a href="https://flink.apache.org/contributing/contribute-documentation.html">https://flink.apache.org/contributing/contribute-documentation.html</a></p>
+ </li>
+ <li>
+ <p>Documentation Style Guide: <a href="https://flink.apache.org/contributing/docs-style.html">https://flink.apache.org/contributing/docs-style.html</a></p>
+ </li>
+</ul>
+
+<p>We look forward to receiving feedback on this GSoD application and also to continue improving the documentation experience for Flink users. Join us!</p>
+</description>
+<pubDate>Mon, 04 May 2020 08:00:00 +0200</pubDate>
+<link>https://flink.apache.org/news/2020/05/04/season-of-docs.html</link>
+<guid isPermaLink="true">/news/2020/05/04/season-of-docs.html</guid>
+</item>
+
+<item>
+<title>Apache Flink 1.9.3 Released</title>
+<description><p>The Apache Flink community released the third bugfix version of the Apache Flink 1.9 series.</p>
+
+<p>This release includes 38 fixes and minor improvements for Flink 1.9.2. The list below includes a detailed list of all fixes and improvements.</p>
+
+<p>We highly recommend all users to upgrade to Flink 1.9.3.</p>
+
+<p>Updated Maven dependencies:</p>
+
+<div class="highlight"><pre><code class="language-xml"><span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-java<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.9.3<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span>
+<span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-streaming-java_2.11<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.9.3<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span>
+<span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-clients_2.11<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.9.3<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span></code></pre></div>
+
+<p>You can find the binaries on the updated <a href="/downloads.html">Downloads page</a>.</p>
+
+<p>List of resolved issues:</p>
+
+<h2> Sub-task
+</h2>
+<ul>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15143">FLINK-15143</a>] - Create document for FLIP-49 TM memory model and configuration guide
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16389">FLINK-16389</a>] - Bump Kafka 0.10 to 0.10.2.2
+</li>
+</ul>
+
+<h2> Bug
+</h2>
+<ul>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11193">FLINK-11193</a>] - Rocksdb timer service factory configuration option is not settable per job
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14316">FLINK-14316</a>] - Stuck in &quot;Job leader ... lost leadership&quot; error
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14560">FLINK-14560</a>] - The value of taskmanager.memory.size in flink-conf.yaml is set to zero will cause taskmanager not to work
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15010">FLINK-15010</a>] - Temp directories flink-netty-shuffle-* are not cleaned up
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15085">FLINK-15085</a>] - HistoryServer dashboard config json out of sync
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15386">FLINK-15386</a>] - SingleJobSubmittedJobGraphStore.putJobGraph has a logic error
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15575">FLINK-15575</a>] - Azure Filesystem Shades Wrong Package &quot;httpcomponents&quot;
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15638">FLINK-15638</a>] - releasing/create_release_branch.sh does not set version in flink-python/pyflink/version.py
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15812">FLINK-15812</a>] - HistoryServer archiving is done in Dispatcher main thread
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15844">FLINK-15844</a>] - Removal of JobWithJars.buildUserCodeClassLoader method without Configuration breaks backwards compatibility
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15863">FLINK-15863</a>] - Fix docs stating that savepoints are relocatable
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16047">FLINK-16047</a>] - Blink planner produces wrong aggregate results with state clean up
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16242">FLINK-16242</a>] - BinaryGeneric serialization error cause checkpoint failure
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16308">FLINK-16308</a>] - SQL connector download links are broken
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16373">FLINK-16373</a>] - EmbeddedLeaderService: IllegalStateException: The RPC connection is already closed
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16573">FLINK-16573</a>] - Kinesis consumer does not properly shutdown RecordFetcher threads
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16576">FLINK-16576</a>] - State inconsistency on restore with memory state backends
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16696">FLINK-16696</a>] - Savepoint trigger documentation is insufficient
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16703">FLINK-16703</a>] - AkkaRpcActor state machine does not record transition to terminating state.
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16836">FLINK-16836</a>] - Losing leadership does not clear rpc connection in JobManagerLeaderListener
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16860">FLINK-16860</a>] - Failed to push filter into OrcTableSource when upgrading to 1.9.2
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16916">FLINK-16916</a>] - The logic of NullableSerializer#copy is wrong
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-17062">FLINK-17062</a>] - Fix the conversion from Java row type to Python row type
+</li>
+</ul>
+
+<h2> Improvement
+</h2>
+<ul>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14278">FLINK-14278</a>] - Pass in ioExecutor into AbstractDispatcherResourceManagerComponentFactory
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15908">FLINK-15908</a>] - Add description of support &#39;pip install&#39; to 1.9.x documents
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15909">FLINK-15909</a>] - Add PyPI release process into the subsequent release of 1.9.x
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15938">FLINK-15938</a>] - Idle state not cleaned in StreamingJoinOperator and StreamingSemiAntiJoinOperator
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16018">FLINK-16018</a>] - Improve error reporting when submitting batch job (instead of AskTimeoutException)
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16031">FLINK-16031</a>] - Improve the description in the README file of PyFlink 1.9.x
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16167">FLINK-16167</a>] - Update documentation about python shell execution
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16280">FLINK-16280</a>] - Fix sample code errors in the documentation about elasticsearch connector
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16697">FLINK-16697</a>] - Disable JMX rebinding
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16862">FLINK-16862</a>] - Remove example url in quickstarts
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16942">FLINK-16942</a>] - ES 5 sink should allow users to select netty transport client
+</li>
+</ul>
+
+<h2> Task
+</h2>
+<ul>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11767">FLINK-11767</a>] - Introduce new TypeSerializerUpgradeTestBase, new PojoSerializerUpgradeTest
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-16454">FLINK-16454</a>] - Update the copyright year in NOTICE files
+</li>
+</ul>
+</description>
+<pubDate>Fri, 24 Apr 2020 14:00:00 +0200</pubDate>
+<link>https://flink.apache.org/news/2020/04/24/release-1.9.3.html</link>
+<guid isPermaLink="true">/news/2020/04/24/release-1.9.3.html</guid>
+</item>
+
+<item>
+<title>Memory Management Improvements with Apache Flink 1.10</title>
+<description><p>Apache Flink 1.10 comes with significant changes to the memory model of the Task Managers and configuration options for your Flink applications. These recently-introduced changes make Flink more adaptable to all kinds of deployment environments (e.g. Kubernetes, Yarn, Mesos), providing strict control over its memory consumption. In this post, we describe Flink’s memory model, as it stands in Flink 1.10, how to set up and manage memory consumption of your Flink appli [...]
+
+<h2 id="introduction-to-flinks-memory-model">Introduction to Flink’s memory model</h2>
+
+<p>Having a clear understanding of Apache Flink’s memory model allows you to manage resources for the various workloads more efficiently. The following diagram illustrates the main memory components in Flink:</p>
+
+<center>
+<img src="/img/blog/2020-04-21-memory-management-improvements-flink-1.10/total-process-memory.svg" width="400px" alt="Flink: Total Process Memory" />
+<br />
+<i><small>Flink: Total Process Memory</small></i>
+</center>
+<p><br /></p>
+
+<p>The Task Manager process is a JVM process. On a high level, its memory consists of the <em>JVM Heap</em> and <em>Off-Heap</em> memory. These types of memory are consumed by Flink directly or by JVM for its specific purposes (i.e. metaspace etc.). There are two major memory consumers within Flink: the user code of job operator tasks and the framework itself consuming memory for internal data structures, network buffers, etc.</p>
+
+<p><strong>Please note that</strong> the user code has direct access to all memory types: <em>JVM Heap, Direct</em> and <em>Native memory</em>. Therefore, Flink cannot really control its allocation and usage. There are however two types of Off-Heap memory which are consumed by tasks and controlled explicitly by Flink:</p>
+
+<ul>
+ <li><em>Managed Memory</em> (Off-Heap)</li>
+ <li><em>Network Buffers</em></li>
+</ul>
+
+<p>The latter is part of the <em>JVM Direct Memory</em>, allocated for user record data exchange between operator tasks.</p>
+
+<h2 id="how-to-set-up-flink-memory">How to set up Flink memory</h2>
+
+<p>With the latest release of Flink 1.10 and in order to provide better user experience, the framework comes with both high-level and fine-grained tuning of memory components. There are essentially three alternatives to setting up memory in Task Managers.</p>
+
+<p>The first two — and simplest — alternatives are configuring one of the two following options for total memory available for the JVM process of the Task Manager:</p>
+
+<ul>
+ <li><em>Total Process Memory</em>: total memory consumed by the Flink Java application (including user code) and by the JVM to run the whole process.</li>
+ <li><em>Total Flink Memory</em>: only memory consumed by the Flink Java application, including user code but excluding memory allocated by JVM to run it</li>
+</ul>
+
+<p>It is advisable to configure the <em>Total Flink Memory</em> for standalone deployments where explicitly declaring how much memory is given to Flink is a common practice, while the outer <em>JVM overhead</em> is of little interest. For the cases of deploying Flink in containerized environments (such as <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/deployment/kubernetes.html">Kubernetes</a>, <a href=&quo [...]
+
+<p>If you want more fine-grained control over the size of <em>JVM Heap</em> and <em>Managed Memory</em> (Off-Heap), there is also a second alternative to configure both <em><a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/memory/mem_setup.html#task-operator-heap-memory">Task Heap</a></em> and <em><a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/memory/mem_setup. [...]
+
+<p>In line with the community’s efforts to <a href="https://flink.apache.org/news/2019/02/13/unified-batch-streaming-blink.html">unify batch and stream processing</a>, this model works universally for both scenarios. It allows sharing the <em>JVM Heap</em> memory between the user code of operator tasks in any workload and the heap state backend in stream processing scenarios. In a similar way, the <em>Managed Memory</em> can be used for [...]
+
+<p>The remaining memory components are automatically adjusted either based on their default values or additionally configured parameters. Flink also checks the overall consistency. You can find more information about the different memory components in the corresponding <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/memory/mem_detail.html">documentation</a>. Additionally, you can try different configuration options with the <a href [...]
+
+<p>If you are migrating from a Flink version older than 1.10, we suggest following the steps in the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/memory/mem_migration.html">migration guide</a> of the Flink documentation.</p>
+
+<h2 id="other-components">Other components</h2>
+
+<p>While configuring Flink’s memory, the size of different memory components can either be fixed with the value of the respective option or tuned using multiple options. Below we provide some more insight about the memory setup.</p>
+
+<h3 id="fractions-of-the-total-flink-memory">Fractions of the Total Flink Memory</h3>
+
+<p>This method allows a proportional breakdown of the <em>Total Flink Memory</em> where the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/memory/mem_setup.html#managed-memory">Managed Memory</a> (if not set explicitly) and <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/memory/mem_detail.html#capped-fractionated-components">Network Buffers</a> can take certain fractions of [...]
+
+<center>
+<img src="/img/blog/2020-04-21-memory-management-improvements-flink-1.10/flink-memory-setup.svg" width="800px" alt="Flink: Example of Memory Setup" />
+<br />
+<i><small>Flink: Example of Memory Setup</small></i>
+</center>
+<p><br /></p>
+
+<p><strong>Please note that</strong></p>
+
+<ul>
+ <li>Flink will verify that the size of the derived <em>Network Memory</em> is between its minimum and maximum value, otherwise Flink’s startup will fail. The maximum and minimum limits have default values which can be overwritten by the respective configuration options.</li>
+ <li>In general, the configured fractions are treated by Flink as hints. Under certain scenarios, the derived value might not match the fraction. For example, if the <em>Total Flink Memory</em> and the <em>Task Heap</em> are configured to fixed values, the <em>Managed Memory</em> will get a certain fraction and the <em>Network Memory</em> will get the remaining memory which might not exactly match its fraction.</li>
+</ul>
+
+<h3 id="more-hints-to-control-the-container-memory-limit">More hints to control the container memory limit</h3>
+
+<p>The heap and direct memory usage are managed by the JVM. There are also many other possible sources of native memory consumption in Apache Flink or its user applications which are not managed by Flink or the JVM. Controlling their limits is often difficult which complicates debugging of potential memory leaks. If Flink’s process allocates too much memory in an unmanaged way, it can often result in killing Task Manager containers in containerized environments. In this case, it ma [...]
+
+<ul>
+ <li>
+ <p><a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/memory/mem_tuning.html#rocksdb-state-backend">RocksDB state cannot grow too big</a>. The memory consumption of RocksDB state backend is accounted for in the <em>Managed Memory</em>. RocksDB respects its limit by default (only since Flink 1.10). You can increase the <em>Managed Memory</em> size to improve RocksDB’s performance or decrease it to save resource [...]
+ </li>
+ <li>
+ <p><a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/memory/mem_setup.html#configure-off-heap-memory-direct-or-native">User code or its dependencies consume significant off-heap memory</a>. Tuning the <em>Task Off-Heap</em> option can assign additional direct or native memory to the user code or any of its dependencies. Flink cannot control native allocations but it sets the limit for <em>JVM Direct</em> memo [...]
+ </li>
+ <li>
+ <p><a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/memory/mem_detail.html#jvm-parameters">JVM metaspace requires additional memory</a>. If you encounter <code>OutOfMemoryError: Metaspace</code>, Flink provides an option to increase its limit and the JVM will ensure that it is not exceeded.</p>
+ </li>
+ <li>
+ <p><a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/memory/mem_detail.html#capped-fractionated-components">JVM requires more internal memory</a>. There is no direct control over certain types of JVM process allocations but Flink provides <em>JVM Overhead</em> options. The options allow declaring an additional amount of memory, anticipated for those allocations and not covered by other options.</p>
+ </li>
+</ul>
+
+<h2 id="conclusion">Conclusion</h2>
+
+<p>The latest Flink release (Flink 1.10) introduces some significant changes to Flink’s memory configuration, making it possible to manage your application memory and debug Flink significantly better than before. Future developments in this area also include adopting a similar memory model for the job manager process in <a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP+116%3A+Unified+Memory+Configuration+for+Job+Managers">FLIP-116</a>, so stay tu [...]
+</description>
+<pubDate>Tue, 21 Apr 2020 14:00:00 +0200</pubDate>
+<link>https://flink.apache.org/news/2020/04/21/memory-management-improvements-flink-1.10.html</link>
+<guid isPermaLink="true">/news/2020/04/21/memory-management-improvements-flink-1.10.html</guid>
+</item>
+
+<item>
+<title>Flink Serialization Tuning Vol. 1: Choosing your Serializer — if you can</title>
+<description><p>Almost every Flink job has to exchange data between its operators and since these records may not only be sent to another instance in the same JVM but instead to a separate process, records need to be serialized to bytes first. Similarly, Flink’s off-heap state-backend is based on a local embedded RocksDB instance which is implemented in native C++ code and thus also needs transformation into bytes on every state access. Wire and state serialization alone can easily [...]
+
+<p>Since serialization is so crucial to your Flink job, we would like to highlight Flink’s serialization stack in a series of blog posts starting with looking at the different ways Flink can serialize your data types.</p>
+
+<div class="page-toc">
+<ul id="markdown-toc">
+ <li><a href="#recap-flink-serialization" id="markdown-toc-recap-flink-serialization">Recap: Flink Serialization</a></li>
+ <li><a href="#choice-of-serializer" id="markdown-toc-choice-of-serializer">Choice of Serializer</a> <ul>
+ <li><a href="#pojoserializer" id="markdown-toc-pojoserializer">PojoSerializer</a></li>
+ <li><a href="#tuple-data-types" id="markdown-toc-tuple-data-types">Tuple Data Types</a></li>
+ <li><a href="#row-data-types" id="markdown-toc-row-data-types">Row Data Types</a></li>
+ <li><a href="#avro" id="markdown-toc-avro">Avro</a> <ul>
+ <li><a href="#avro-specific" id="markdown-toc-avro-specific">Avro Specific</a></li>
+ <li><a href="#avro-generic" id="markdown-toc-avro-generic">Avro Generic</a></li>
+ <li><a href="#avro-reflect" id="markdown-toc-avro-reflect">Avro Reflect</a></li>
+ </ul>
+ </li>
+ <li><a href="#kryo" id="markdown-toc-kryo">Kryo</a> <ul>
+ <li><a href="#disabling-kryo" id="markdown-toc-disabling-kryo">Disabling Kryo</a></li>
+ </ul>
+ </li>
+ <li><a href="#apache-thrift-via-kryo" id="markdown-toc-apache-thrift-via-kryo">Apache Thrift (via Kryo)</a></li>
+ <li><a href="#protobuf-via-kryo" id="markdown-toc-protobuf-via-kryo">Protobuf (via Kryo)</a></li>
+ </ul>
+ </li>
+ <li><a href="#state-schema-evolution" id="markdown-toc-state-schema-evolution">State Schema Evolution</a></li>
+ <li><a href="#performance-comparison" id="markdown-toc-performance-comparison">Performance Comparison</a></li>
+ <li><a href="#conclusion" id="markdown-toc-conclusion">Conclusion</a></li>
+</ul>
+
+</div>
+
+<h1 id="recap-flink-serialization">Recap: Flink Serialization</h1>
+
+<p>Flink handles <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/types_serialization.html">data types and serialization</a> with its own type descriptors, generic type extraction, and type serialization framework. We recommend reading through the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/types_serialization.html">documentation</a> first in order to be able to follow the arguments w [...]
+<code>stream.keyBy(“ruleId”)</code> or
+<code>dataSet.join(another).where("name").equalTo("personName")</code>. It also allows optimizations in the serialization format as well as reducing unnecessary de/serializations (mainly in certain Batch operations as well as in the SQL/Table APIs).</p>
+
+<h1 id="choice-of-serializer">Choice of Serializer</h1>
+
+<p>Apache Flink’s out-of-the-box serialization can be roughly divided into the following groups:</p>
+
+<ul>
+ <li>
+ <p><strong>Flink-provided special serializers</strong> for basic types (Java primitives and their boxed form), arrays, composite types (tuples, Scala case classes, Rows), and a few auxiliary types (Option, Either, Lists, Maps, …),</p>
+ </li>
+ <li>
+ <p><strong>POJOs</strong>; a public, standalone class with a public no-argument constructor and all non-static, non-transient fields in the class hierarchy either public or with a public getter- and a setter-method; see <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/types_serialization.html#rules-for-pojo-types">POJO Rules</a>,</p>
+ </li>
+ <li>
+ <p><strong>Generic types</strong>; user-defined data types that are not recognized as a POJO and then serialized via <a href="https://github.com/EsotericSoftware/kryo">Kryo</a>.</p>
+ </li>
+</ul>
+
+<p>Alternatively, you can also register <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/custom_serializers.html">custom serializers</a> for user-defined data types. This includes writing your own serializers or integrating other serialization systems like <a href="https://developers.google.com/protocol-buffers/">Google Protobuf</a> or <a href="https://thrift.apache.org/">Apache Thrift</a> vi [...]
+
+<h2 id="pojoserializer">PojoSerializer</h2>
+
+<p>As outlined above, if your data type is not covered by a specialized serializer but follows the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/types_serialization.html#rules-for-pojo-types">POJO Rules</a>, it will be serialized with the <a href="https://github.com/apache/flink/blob/release-1.10.0/flink-core/src/main/java/org/apache/flink/api/java/typeutils/runtime/PojoSerializer.java">PojoSerializer</a> which [...]
+
+<blockquote>
+ <p>15:45:51,460 INFO org.apache.flink.api.java.typeutils.TypeExtractor - Class … cannot be used as a POJO type because not all fields are valid POJO fields, and must be processed as GenericType. Please read the Flink documentation on “Data Types &amp; Serialization” for details of the effect on performance.</p>
+</blockquote>
+
+<p>This means, that the PojoSerializer will not be used, but instead Flink will fall back to Kryo for serialization (see below). We will have a more detailed look into a few (more) situations that can lead to unexpected Kryo fallbacks in the second part of this blog post series.</p>
+
+<h2 id="tuple-data-types">Tuple Data Types</h2>
+
+<p>Flink comes with a predefined set of tuple types which all have a fixed length and contain a set of strongly-typed fields of potentially different types. There are implementations for <code>Tuple0</code>, <code>Tuple1&lt;T0&gt;</code>, …, <code>Tuple25&lt;T0, T1, ..., T24&gt;</code> and they may serve as easy-to-use wrappers that spare the creation of POJOs for each and every combination of objects you need to pass between comp [...]
+
+<div class="alert alert-info">
+ <p><span class="label label-info" style="display: inline-block"><span class="glyphicon glyphicon-info-sign" aria-hidden="true"></span> Note</span>
+ Since <code>Tuple0</code> does not contain any data and therefore is probably a bit special anyway, it will use a special serializer implementation: <a href="https://github.com/apache/flink/blob/release-1.10.0/flink-core/src/main/java/org/apache/flink/api/java/typeutils/runtime/Tuple0Serializer.java">Tuple0Serializer</a>.</p>
+</div>
+
+<h2 id="row-data-types">Row Data Types</h2>
+
+<p>Row types are mainly used by the Table and SQL APIs of Flink. A <code>Row</code> groups an arbitrary number of objects together similar to the tuples above. These fields are not strongly typed and may all be of different types. Because field types are missing, Flink’s type extraction cannot automatically extract type information and users of a <code>Row</code> need to manually tell Flink about the row’s field types. The <a href="https://github.com [...]
+
+<p>Row type information can be provided in two ways:</p>
+
+<ul>
+ <li>you can have your source or operator implement <code>ResultTypeQueryable&lt;Row&gt;</code>:</li>
+</ul>
+
+<div class="highlight"><pre><code class="language-java"><span class="kd">public</span> <span class="kd">static</span> <span class="kd">class</span> <span class="nc">RowSource</span> <span class="kd">implements</span> <span class="n">SourceFunction</span><span class="o">&lt;</span><span class= [...]
+ <span class="c1">// ...</span>
+
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="n">TypeInformation</span><span class="o">&lt;</span><span class="n">Row</span><span class="o">&gt;</span> <span class="nf">getProducedType</span><span class="o">()</span> <span class="o">{</span>
+ <span class="k">return</span> <span class="n">Types</span><span class="o">.</span><span class="na">ROW</span><span class="o">(</span><span class="n">Types</span><span class="o">.</span><span class="na">INT</span><span class="o">,</span> <span class="n">Types</span>< [...]
+ <span class="o">}</span>
+<span class="o">}</span></code></pre></div>
+
+<ul>
+ <li>you can provide the types when building the job graph by using <code>SingleOutputStreamOperator#returns()</code></li>
+</ul>
+
+<div class="highlight"><pre><code class="language-java"><span class="n">DataStream</span><span class="o">&lt;</span><span class="n">Row</span><span class="o">&gt;</span> <span class="n">sourceStream</span> <span class="o">=</span>
+ <span class="n">env</span><span class="o">.</span><span class="na">addSource</span><span class="o">(</span><span class="k">new</span> <span class="nf">RowSource</span><span class="o">())</span>
+ <span class="o">.</span><span class="na">returns</span><span class="o">(</span><span class="n">Types</span><span class="o">.</span><span class="na">ROW</span><span class="o">(</span><span class="n">Types</span><span class="o">.</span><span class="na">INT</span>< [...]
+
+<div class="alert alert-warning">
+ <p><span class="label label-warning" style="display: inline-block"><span class="glyphicon glyphicon-warning-sign" aria-hidden="true"></span> Warning</span>
+If you fail to provide the type information for a <code>Row</code>, Flink identifies that <code>Row</code> is not a valid POJO type according to the rules above and falls back to Kryo serialization (see below) which you will also see in the logs as:</p>
+
+ <p><code>13:10:11,148 INFO org.apache.flink.api.java.typeutils.TypeExtractor - Class class org.apache.flink.types.Row cannot be used as a POJO type because not all fields are valid POJO fields, and must be processed as GenericType. Please read the Flink documentation on "Data Types &amp; Serialization" for details of the effect on performance.</code></p>
+</div>
+
+<h2 id="avro">Avro</h2>
+
+<p>Flink offers built-in support for the <a href="http://avro.apache.org/">Apache Avro</a> serialization framework (currently using version 1.8.2) by adding the <code>org.apache.flink:flink-avro</code> dependency into your job. Flink’s <a href="https://github.com/apache/flink/blob/release-1.10.0/flink-formats/flink-avro/src/main/java/org/apache/flink/formats/avro/typeutils/AvroSerializer.java">AvroSerializer</a> can then use A [...]
+
+<h3 id="avro-specific">Avro Specific</h3>
+
+<p>Avro specific records will be automatically detected by checking that the given type’s type hierarchy contains the <code>SpecificRecordBase</code> class. You can either specify your concrete Avro type, or—if you want to be more generic and allow different types in your operator—use the <code>SpecificRecordBase</code> type (or a subtype) in your user functions, in <code>ResultTypeQueryable#getProducedType()</code>, or in <code>SingleOutpu [...]
+
+<div class="alert alert-warning">
+ <p><span class="label label-warning" style="display: inline-block"><span class="glyphicon glyphicon-warning-sign" aria-hidden="true"></span> Warning</span> If you specify the Flink type as <code>SpecificRecord</code> and not <code>SpecificRecordBase</code>, Flink will not see this as an Avro type. Instead, it will use Kryo to de/serialize any objects which may be considerably slower.</p>
+</div>
+
+<h3 id="avro-generic">Avro Generic</h3>
+
+<p>Avro’s <code>GenericRecord</code> types cannot, unfortunately, be used automatically since they require the user to <a href="https://avro.apache.org/docs/1.8.2/gettingstartedjava.html#Serializing+and+deserializing+without+code+generation">specify a schema</a> (either manually or by retrieving it from some schema registry). With that schema, you can provide the right type information by either of the following options just like for the Row Types [...]
+
+<ul>
+ <li>implement <code>ResultTypeQueryable&lt;GenericRecord&gt;</code>:</li>
+</ul>
+
+<div class="highlight"><pre><code class="language-java"><span class="kd">public</span> <span class="kd">static</span> <span class="kd">class</span> <span class="nc">AvroGenericSource</span> <span class="kd">implements</span> <span class="n">SourceFunction</span><span class="o">&lt;</span><spa [...]
+ <span class="kd">private</span> <span class="kd">final</span> <span class="n">GenericRecordAvroTypeInfo</span> <span class="n">producedType</span><span class="o">;</span>
+
+ <span class="kd">public</span> <span class="nf">AvroGenericSource</span><span class="o">(</span><span class="n">Schema</span> <span class="n">schema</span><span class="o">)</span> <span class="o">{</span>
+ <span class="k">this</span><span class="o">.</span><span class="na">producedType</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">GenericRecordAvroTypeInfo</span><span class="o">(</span><span class="n">schema</span><span class="o">);</span>
+ <span class="o">}</span>
+
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="n">TypeInformation</span><span class="o">&lt;</span><span class="n">GenericRecord</span><span class="o">&gt;</span> <span class="nf">getProducedType</span><span class="o">()</span> <span class="o">{</span>
+ <span class="k">return</span> <span class="n">producedType</span><span class="o">;</span>
+ <span class="o">}</span>
+<span class="o">}</span></code></pre></div>
+<ul>
+ <li>provide type information when building the job graph by using <code>SingleOutputStreamOperator#returns()</code></li>
+</ul>
+
+<div class="highlight"><pre><code class="language-java"><span class="n">DataStream</span><span class="o">&lt;</span><span class="n">GenericRecord</span><span class="o">&gt;</span> <span class="n">sourceStream</span> <span class="o">=</span>
+ <span class="n">env</span><span class="o">.</span><span class="na">addSource</span><span class="o">(</span><span class="k">new</span> <span class="nf">AvroGenericSource</span><span class="o">())</span>
+ <span class="o">.</span><span class="na">returns</span><span class="o">(</span><span class="k">new</span> <span class="nf">GenericRecordAvroTypeInfo</span><span class="o">(</span><span class="n">schema</span><span class="o">));</span></code></pre></div>
+<p>Without this type information, Flink will fall back to Kryo for serialization which would serialize the schema into every record, over and over again. As a result, the serialized form will be bigger and more costly to create.</p>
+
+<div class="alert alert-info">
+ <p><span class="label label-info" style="display: inline-block"><span class="glyphicon glyphicon-info-sign" aria-hidden="true"></span> Note</span>
+ Since Avro’s <code>Schema</code> class is not serializable, it can not be sent around as is. You can work around this by converting it to a String and parsing it back when needed. If you only do this once on initialization, there is practically no difference to sending it directly.</p>
+</div>
+
+<h3 id="avro-reflect">Avro Reflect</h3>
+
+<p>The third way of using Avro is to exchange Flink’s PojoSerializer (for POJOs according to the rules above) for Avro’s reflection-based serializer. This can be enabled by calling</p>
+
+<div class="highlight"><pre><code class="language-java"><span class="n">env</span><span class="o">.</span><span class="na">getConfig</span><span class="o">().</span><span class="na">enableForceAvro</span><span class="o">();</span></code></pre></div>
+
+<h2 id="kryo">Kryo</h2>
+
+<p>Any class or object which does not fall into the categories above or is covered by a Flink-provided special serializer is de/serialized with a fallback to <a href="https://github.com/EsotericSoftware/kryo">Kryo</a> (currently version 2.24.0) which is a powerful and generic serialization framework in Java. Flink calls such a type a <em>generic type</em> and you may stumble upon <code>GenericTypeInfo</code> when debugging code. If you [...]
+
+<div class="highlight"><pre><code class="language-java"><span class="n">env</span><span class="o">.</span><span class="na">getConfig</span><span class="o">().</span><span class="na">registerKryoType</span><span class="o">(</span><span class="n">MyCustomType</span><span class="o">.< [...]
+<p>Registering types adds them to an internal map of classes to tags so that, during serialization, Kryo does not have to add the fully qualified class names as a prefix into the serialized form. Instead, Kryo uses these (integer) tags to identify the underlying classes and reduce serialization overhead.</p>
+
+<div class="alert alert-info">
+ <p><span class="label label-info" style="display: inline-block"><span class="glyphicon glyphicon-info-sign" aria-hidden="true"></span> Note</span>
+Flink will store Kryo serializer mappings from type registrations in its checkpoints and savepoints and will retain them across job (re)starts.</p>
+</div>
+
+<h3 id="disabling-kryo">Disabling Kryo</h3>
+
+<p>If desired, you can disable the Kryo fallback, i.e. the ability to serialize generic types, by calling</p>
+
+<div class="highlight"><pre><code class="language-java"><span class="n">env</span><span class="o">.</span><span class="na">getConfig</span><span class="o">().</span><span class="na">disableGenericTypes</span><span class="o">();</span></code></pre></div>
+
+<p>This is mostly useful for finding out where these fallbacks are applied and replacing them with better serializers. If your job has any generic types with this configuration, it will fail with</p>
+
+<blockquote>
+ <p>Exception in thread “main” java.lang.UnsupportedOperationException: Generic types have been disabled in the ExecutionConfig and type … is treated as a generic type.</p>
+</blockquote>
+
+<p>If you cannot immediately see from the type where it is being used, this log message also gives you a stacktrace that can be used to set breakpoints and find out more details in your IDE.</p>
+
+<h2 id="apache-thrift-via-kryo">Apache Thrift (via Kryo)</h2>
+
+<p>In addition to the variants above, Flink also allows you to <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/custom_serializers.html#register-a-custom-serializer-for-your-flink-program">register other type serialization frameworks</a> with Kryo. After adding the appropriate dependencies from the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/custom_serializers.html#register-a-custom-serializer-for- [...]
+
+<div class="highlight"><pre><code class="language-java"><span class="n">env</span><span class="o">.</span><span class="na">getConfig</span><span class="o">().</span><span class="na">addDefaultKryoSerializer</span><span class="o">(</span><span class="n">MyCustomType</span><span class="o" [...]
+
+<p>This only works if generic types are not disabled and <code>MyCustomType</code> is a Thrift-generated data type. If the data type is not generated by Thrift, Flink will fail at runtime with an exception like this:</p>
+
+<blockquote>
+ <p>java.lang.ClassCastException: class MyCustomType cannot be cast to class org.apache.thrift.TBase (MyCustomType and org.apache.thrift.TBase are in unnamed module of loader ‘app’)</p>
+</blockquote>
+
+<div class="alert alert-info">
+ <p><span class="label label-info" style="display: inline-block"><span class="glyphicon glyphicon-info-sign" aria-hidden="true"></span> Note</span>
+Please note that <code>TBaseSerializer</code> can be registered as a default Kryo serializer as above (and as specified in <a href="https://github.com/twitter/chill/blob/v0.7.6/chill-thrift/src/main/java/com/twitter/chill/thrift/TBaseSerializer.java">its documentation</a>) or via <code>registerTypeWithKryoSerializer</code>. In practice, we found both ways working. We also saw no difference between registering Thrift classes in addition to the [...]
+</div>
+
+<h2 id="protobuf-via-kryo">Protobuf (via Kryo)</h2>
+
+<p>In a way similar to Apache Thrift, <a href="https://developers.google.com/protocol-buffers/">Google Protobuf</a> may be <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/custom_serializers.html#register-a-custom-serializer-for-your-flink-program">registered as a custom serializer</a> after adding the right dependencies (<code>com.twitter:chill-protobuf</code> and <code>com.google.protobuf:proto [...]
+
+<div class="highlight"><pre><code class="language-java"><span class="n">env</span><span class="o">.</span><span class="na">getConfig</span><span class="o">().</span><span class="na">registerTypeWithKryoSerializer</span><span class="o">(</span><span class="n">MyCustomType</span><span class="o [...]
+<p>This will work as long as generic types have not been disabled (this would disable Kryo for good). If <code>MyCustomType</code> is not a Protobuf-generated class, your Flink job will fail at runtime with the following exception:</p>
+
+<blockquote>
+ <p>java.lang.ClassCastException: class <code>MyCustomType</code> cannot be cast to class com.google.protobuf.Message (<code>MyCustomType</code> and com.google.protobuf.Message are in unnamed module of loader ‘app’)</p>
+</blockquote>
+
+<div class="alert alert-info">
+ <p><span class="label label-info" style="display: inline-block"><span class="glyphicon glyphicon-info-sign" aria-hidden="true"></span> Note</span>
+Please note that <code>ProtobufSerializer</code> can be registered as a default Kryo serializer (as specified in the <a href="https://github.com/twitter/chill/blob/v0.7.6/chill-thrift/src/main/java/com/twitter/chill/thrift/TBaseSerializer.java">Protobuf documentation</a>) or via <code>registerTypeWithKryoSerializer</code> (as presented here). In practice, we found both ways working. We also saw no difference between registering your Protobuf [...]
+</div>
+
+<h1 id="state-schema-evolution">State Schema Evolution</h1>
+
+<p>Before taking a closer look at the performance of each of the serializers described above, we would like to emphasize that performance is not everything that counts inside a real-world Flink job. Types for storing state, for example, should be able to evolve their schema (add/remove/change fields) throughout the lifetime of the job without losing previous state. This is what Flink calls <a href="https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/sche [...]
+
+<h1 id="performance-comparison">Performance Comparison</h1>
+
+<p>With so many options for serialization, it is actually not easy to make the right choice. We already saw some technical advantages and disadvantages of each of them outlined above. Since serializers are at the core of your Flink jobs and usually also sit on the hot path (per record invocations), let us actually take a deeper look into their performance with the help of the Flink benchmarks project at <a href="https://github.com/dataArtisans/flink-benchmarks">http [...]
+
+<div class="highlight"><pre><code class="language-java"><span class="kd">public</span> <span class="kd">class</span> <span class="nc">MyPojo</span> <span class="o">{</span>
+ <span class="kd">public</span> <span class="kt">int</span> <span class="n">id</span><span class="o">;</span>
+ <span class="kd">private</span> <span class="n">String</span> <span class="n">name</span><span class="o">;</span>
+ <span class="kd">private</span> <span class="n">String</span><span class="o">[]</span> <span class="n">operationNames</span><span class="o">;</span>
+ <span class="kd">private</span> <span class="n">MyOperation</span><span class="o">[]</span> <span class="n">operations</span><span class="o">;</span>
+ <span class="kd">private</span> <span class="kt">int</span> <span class="n">otherId1</span><span class="o">;</span>
+ <span class="kd">private</span> <span class="kt">int</span> <span class="n">otherId2</span><span class="o">;</span>
+ <span class="kd">private</span> <span class="kt">int</span> <span class="n">otherId3</span><span class="o">;</span>
+ <span class="kd">private</span> <span class="n">Object</span> <span class="n">someObject</span><span class="o">;</span>
+<span class="o">}</span>
+<span class="kd">public</span> <span class="kd">class</span> <span class="nc">MyOperation</span> <span class="o">{</span>
+ <span class="kt">int</span> <span class="n">id</span><span class="o">;</span>
+ <span class="kd">protected</span> <span class="n">String</span> <span class="n">name</span><span class="o">;</span>
+<span class="o">}</span></code></pre></div>
+
+<p>This is mapped to tuples, rows, Avro specific records, Thrift and Protobuf representations appropriately and sent through a simple Flink job at parallelism 4 where the data type is used during network communication like this:</p>
+
+<div class="highlight"><pre><code class="language-java"><span class="n">env</span><span class="o">.</span><span class="na">setParallelism</span><span class="o">(</span><span class="mi">4</span><span class="o">);</span>
+<span class="n">env</span><span class="o">.</span><span class="na">addSource</span><span class="o">(</span><span class="k">new</span> <span class="nf">PojoSource</span><span class="o">(</span><span class="n">RECORDS_PER_INVOCATION</span><span class="o">,</span> <span class="mi"> [...]
+ <span class="o">.</span><span class="na">rebalance</span><span class="o">()</span>
+ <span class="o">.</span><span class="na">addSink</span><span class="o">(</span><span class="k">new</span> <span class="n">DiscardingSink</span><span class="o">&lt;&gt;());</span></code></pre></div>
+<p>After running this through the <a href="http://openjdk.java.net/projects/code-tools/jmh/">jmh</a> micro-benchmarks defined in <a href="https://github.com/dataArtisans/flink-benchmarks/blob/master/src/main/java/org/apache/flink/benchmark/full/SerializationFrameworkAllBenchmarks.java">SerializationFrameworkAllBenchmarks.java</a>, I retrieved the following performance results for Flink 1.10 on my machine (in number of operations per milli [...]
+<br /></p>
+
+<center>
+<img src="/img/blog/2020-04-15-flink-serialization-performance-results.svg" width="800px" alt="Communication between the Flink operator and the Python execution environment" />
+</center>
+<p><br /></p>
+
+<p>A few takeaways from these numbers:</p>
+
+<ul>
+ <li>
+ <p>The default fallback from POJO to Kryo reduces performance by 75%.<br />
+Registering types with Kryo significantly improves its performance with only 64% fewer operations than by using a POJO.</p>
+ </li>
+ <li>
+ <p>Avro GenericRecord and SpecificRecord are roughly serialized at the same speed.</p>
+ </li>
+ <li>
+ <p>Avro Reflect serialization is even slower than Kryo default (-45%).</p>
+ </li>
+ <li>
+ <p>Tuples are the fastest, closely followed by Rows. Both leverage fast specialized serialization code based on direct access without Java reflection.</p>
+ </li>
+ <li>
+ <p>Using a (nested) Tuple instead of a POJO may speed up your job by 42% (but is less flexible!).
+ Having code-generation for the PojoSerializer (<a href="https://jira.apache.org/jira/browse/FLINK-3599">FLINK-3599</a>) may actually close that gap (or at least move closer to the RowSerializer). If you feel like giving the implementation a go, please give the Flink community a note and we will see whether we can make that happen.</p>
+ </li>
+ <li>
+ <p>If you cannot use POJOs, try to define your data type with one of the serialization frameworks that generate specific code for it: Protobuf, Avro, Thrift (in that order, performance-wise).</p>
+ </li>
+</ul>
+
+<div class="alert alert-info">
+ <p><span class="label label-info" style="display: inline-block"><span class="glyphicon glyphicon-info-sign" aria-hidden="true"></span> Note</span> As with all benchmarks, please bear in mind that these numbers only give a hint on Flink’s serializer performance in a specific scenario. They may be different with your data types but the rough classification is probably the same. If you want to be sure, please verify the [...]
+</div>
+
+<h1 id="conclusion">Conclusion</h1>
+
+<p>In the sections above, we looked at how Flink performs serialization for different sorts of data types and elaborated the technical advantages and disadvantages. For data types used in Flink state, you probably want to leverage either POJO or Avro types which, currently, are the only ones supporting state evolution out of the box and allow your stateful application to develop over time. POJOs are usually faster in the de/serialization while Avro may support more flexible schema [...]
+
+<p>The fastest de/serialization is achieved with Flink’s internal tuple and row serializers which can access these types’ fields directly without going via reflection. With roughly 30% decreased throughput as compared to tuples, Protobuf and POJO types do not perform too badly on their own and are more flexible and maintainable. Avro (specific and generic) records as well as Thrift data types further reduce performance by 20% and 30%, respectively. You definitely want to avoid Kryo [...]
+
+<p>The next article in this series will use this finding as a starting point to look into a few common pitfalls and obstacles of avoiding Kryo, how to get the most out of the PojoSerializer, and a few more tuning techniques with respect to serialization. Stay tuned for more.</p>
+</description>
+<pubDate>Wed, 15 Apr 2020 10:00:00 +0200</pubDate>
+<link>https://flink.apache.org/news/2020/04/15/flink-serialization-tuning-vol-1.html</link>
+<guid isPermaLink="true">/news/2020/04/15/flink-serialization-tuning-vol-1.html</guid>
+</item>
+
+<item>
+<title>PyFlink: Introducing Python Support for UDFs in Flink's Table API</title>
+<description><p>Flink 1.9 introduced the Python Table API, allowing developers and data engineers to write Python Table API jobs for Table transformations and analysis, such as Python ETL or aggregate jobs. However, Python users faced some limitations when it came to support for Python UDFs in Flink 1.9, preventing them from extending the system’s built-in functionality.</p>
+
+<p>In Flink 1.10, the community further extended the support for Python by adding Python UDFs in PyFlink. Additionally, both the Python UDF environment and dependency management are now supported, allowing users to import third-party libraries in the UDFs, leveraging Python’s rich set of third-party libraries.</p>
+
+<h1 id="python-support-for-udfs-in-flink-110">Python Support for UDFs in Flink 1.10</h1>
+
+<p>Before diving into how you can define and use Python UDFs, we explain the motivation and background behind how UDFs work in PyFlink and provide some additional context about the implementation of our approach. Below we give a brief introduction on the PyFlink architecture from job submission, all the way to executing the Python UDF.</p>
+
+<p>The PyFlink architecture mainly includes two parts — local and cluster — as shown in the architecture visual below. The local phase is the compilation of the job, and the cluster is the execution of the job.</p>
+
+<center>
+<img src="/img/blog/2020-04-09-pyflink-udfs/pyflink-udf-architecture.png" width="600px" alt="PyFlink UDF Architecture" />
+</center>
+<p><br /></p>
+
+<p>For the local part, the Python API is a mapping of the Java API: each time Python executes a method in the figure above, it will synchronously call the method corresponding to Java through Py4J, and finally generate a Java JobGraph, before submitting it to the cluster.</p>
+
+<p>For the cluster part, just like ordinary Java jobs, the JobMaster schedules tasks to TaskManagers. The tasks that include Python UDF in a TaskManager involve the execution of Java and Python operators. In the Python UDF operator, various gRPC services are used to provide different communications between the Java VM and the Python VM, such as DataService for data transmissions, StateService for state requirements, and Logging and Metrics Services. These services are built on Beam [...]
+
+<h1 id="how-to-use-pyflink-with-udfs-in-flink-110">How to use PyFlink with UDFs in Flink 1.10</h1>
+
+<p>This section provides some Python user defined function (UDF) examples, including how to install PyFlink, how to define/register/invoke UDFs in PyFlink and how to execute the job.</p>
+
+<h2 id="install-pyflink">Install PyFlink</h2>
+<p>Using Python in Apache Flink requires installing PyFlink. PyFlink is available through PyPI and can be easily installed using pip:</p>
+
+<div class="highlight"><pre><code class="language-bash"><span class="nv">$ </span>python -m pip install apache-flink</code></pre></div>
+<div class="alert alert-info">
+ <p><span class="label label-info" style="display: inline-block"><span class="glyphicon glyphicon-info-sign" aria-hidden="true"></span> Note</span>
+Please note that Python 3.5 or higher is required to install and run PyFlink</p>
+</div>
+
+<p><br /></p>
+
+<h2 id="define-a-python-udf">Define a Python UDF</h2>
+
+<p>There are many ways to define a Python scalar function, besides extending the base class <code>ScalarFunction</code>. The following example shows the different ways of defining a Python scalar function that takes two columns of <code>BIGINT</code> as input parameters and returns the sum of them as the result.</p>
+
+<div class="highlight"><pre><code class="language-python"><span class="c"># option 1: extending the base class `ScalarFunction`</span>
+<span class="k">class</span> <span class="nc">Add</span><span class="p">(</span><span class="n">ScalarFunction</span><span class="p">):</span>
+ <span class="k">def</span> <span class="nf">eval</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">i</span><span class="p">,</span> <span class="n">j</span><span class="p">):</span>
+ <span class="k">return</span> <span class="n">i</span> <span class="o">+</span> <span class="n">j</span>
+
+<span class="n">add</span> <span class="o">=</span> <span class="n">udf</span><span class="p">(</span><span class="n">Add</span><span class="p">(),</span> <span class="p">[</span><span class="n">DataTypes</span><span class="o">.</span><span class="n">BIGINT</span><span [...]
+
+<span class="c"># option 2: Python function</span>
+<span class="nd">@udf</span><span class="p">(</span><span class="n">input_types</span><span class="o">=</span><span class="p">[</span><span class="n">DataTypes</span><span class="o">.</span><span class="n">BIGINT</span><span class="p">(),</span> <span class="n">DataTypes</sp [...]
+<span class="k">def</span> <span class="nf">add</span><span class="p">(</span><span class="n">i</span><span class="p">,</span> <span class="n">j</span><span class="p">):</span>
+ <span class="k">return</span> <span class="n">i</span> <span class="o">+</span> <span class="n">j</span>
+
+<span class="c"># option 3: lambda function</span>
+<span class="n">add</span> <span class="o">=</span> <span class="n">udf</span><span class="p">(</span><span class="k">lambda</span> <span class="n">i</span><span class="p">,</span> <span class="n">j</span><span class="p">:</span> <span class="n">i</span> <span class=&q [...]
+
+<span class="c"># option 4: callable function</span>
+<span class="k">class</span> <span class="nc">CallableAdd</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
+ <span class="k">def</span> <span class="nf">__call__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">i</span><span class="p">,</span> <span class="n">j</span><span class="p">):</span>
+ <span class="k">return</span> <span class="n">i</span> <span class="o">+</span> <span class="n">j</span>
+
+<span class="n">add</span> <span class="o">=</span> <span class="n">udf</span><span class="p">(</span><span class="n">CallableAdd</span><span class="p">(),</span> <span class="p">[</span><span class="n">DataTypes</span><span class="o">.</span><span class="n">BIGINT</span> [...]
+
+<span class="c"># option 5: partial function</span>
+<span class="k">def</span> <span class="nf">partial_add</span><span class="p">(</span><span class="n">i</span><span class="p">,</span> <span class="n">j</span><span class="p">,</span> <span class="n">k</span><span class="p">):</span>
+ <span class="k">return</span> <span class="n">i</span> <span class="o">+</span> <span class="n">j</span> <span class="o">+</span> <span class="n">k</span>
+
+<span class="n">add</span> <span class="o">=</span> <span class="n">udf</span><span class="p">(</span><span class="n">functools</span><span class="o">.</span><span class="n">partial</span><span class="p">(</span><span class="n">partial_add</span><span class="p">,</span> & [...]
+ <span class="n">DataTypes</span><span class="o">.</span><span class="n">BIGINT</span><span class="p">())</span></code></pre></div>
+
+<h2 id="register-a-python-udf">Register a Python UDF</h2>
+
+<div class="highlight"><pre><code class="language-python"><span class="c"># register the Python function</span>
+<span class="n">table_env</span><span class="o">.</span><span class="n">register_function</span><span class="p">(</span><span class="s">&quot;add&quot;</span><span class="p">,</span> <span class="n">add</span><span class="p">)</span></code></pre></div>
+
+<h2 id="invoke-a-python-udf">Invoke a Python UDF</h2>
+
+<div class="highlight"><pre><code class="language-python"><span class="c"># use the function in Python Table API</span>
+<span class="n">my_table</span><span class="o">.</span><span class="n">select</span><span class="p">(</span><span class="s">&quot;add(a, b)&quot;</span><span class="p">)</span></code></pre></div>
+
+<p>Below, you can find a complete example of using Python UDF.</p>
+
+<div class="highlight"><pre><code class="language-python"><span class="kn">from</span> <span class="nn">pyflink.datastream</span> <span class="kn">import</span> <span class="n">StreamExecutionEnvironment</span>
+<span class="kn">from</span> <span class="nn">pyflink.table</span> <span class="kn">import</span> <span class="n">StreamTableEnvironment</span><span class="p">,</span> <span class="n">DataTypes</span>
+<span class="kn">from</span> <span class="nn">pyflink.table.descriptors</span> <span class="kn">import</span> <span class="n">Schema</span><span class="p">,</span> <span class="n">OldCsv</span><span class="p">,</span> <span class="n">FileSystem</span>
+<span class="kn">from</span> <span class="nn">pyflink.table.udf</span> <span class="kn">import</span> <span class="n">udf</span>
+
+<span class="n">env</span> <span class="o">=</span> <span class="n">StreamExecutionEnvironment</span><span class="o">.</span><span class="n">get_execution_environment</span><span class="p">()</span>
+<span class="n">env</span><span class="o">.</span><span class="n">set_parallelism</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
+<span class="n">t_env</span> <span class="o">=</span> <span class="n">StreamTableEnvironment</span><span class="o">.</span><span class="n">create</span><span class="p">(</span><span class="n">env</span><span class="p">)</span>
+
+<span class="n">add</span> <span class="o">=</span> <span class="n">udf</span><span class="p">(</span><span class="k">lambda</span> <span class="n">i</span><span class="p">,</span> <span class="n">j</span><span class="p">:</span> <span class="n">i</span> <span class=&q [...]
+
+<span class="n">t_env</span><span class="o">.</span><span class="n">register_function</span><span class="p">(</span><span class="s">&quot;add&quot;</span><span class="p">,</span> <span class="n">add</span><span class="p">)</span>
+
+<span class="n">t_env</span><span class="o">.</span><span class="n">connect</span><span class="p">(</span><span class="n">FileSystem</span><span class="p">()</span><span class="o">.</span><span class="n">path</span><span class="p">(</span><span class="s">&#39;/tmp/input& [...]
+ <span class="o">.</span><span class="n">with_format</span><span class="p">(</span><span class="n">OldCsv</span><span class="p">()</span>
+ <span class="o">.</span><span class="n">field</span><span class="p">(</span><span class="s">&#39;a&#39;</span><span class="p">,</span> <span class="n">DataTypes</span><span class="o">.</span><span class="n">BIGINT</span><span class="p">())</span>
+ <span class="o">.</span><span class="n">field</span><span class="p">(</span><span class="s">&#39;b&#39;</span><span class="p">,</span> <span class="n">DataTypes</span><span class="o">.</span><span class="n">BIGINT</span><span class="p">()))</span> \
+ <span class="o">.</span><span class="n">with_schema</span><span class="p">(</span><span class="n">Schema</span><span class="p">()</span>
+ <span class="o">.</span><span class="n">field</span><span class="p">(</span><span class="s">&#39;a&#39;</span><span class="p">,</span> <span class="n">DataTypes</span><span class="o">.</span><span class="n">BIGINT</span><span class="p">())</span>
+ <span class="o">.</span><span class="n">field</span><span class="p">(</span><span class="s">&#39;b&#39;</span><span class="p">,</span> <span class="n">DataTypes</span><span class="o">.</span><span class="n">BIGINT</span><span class="p">()))</span> \
+ <span class="o">.</span><span class="n">create_temporary_table</span><span class="p">(</span><span class="s">&#39;mySource&#39;</span><span class="p">)</span>
+
+<span class="n">t_env</span><span class="o">.</span><span class="n">connect</span><span class="p">(</span><span class="n">FileSystem</span><span class="p">()</span><span class="o">.</span><span class="n">path</span><span class="p">(</span><span class="s">&#39;/tmp/output&am [...]
+ <span class="o">.</span><span class="n">with_format</span><span class="p">(</span><span class="n">OldCsv</span><span class="p">()</span>
+ <span class="o">.</span><span class="n">field</span><span class="p">(</span><span class="s">&#39;sum&#39;</span><span class="p">,</span> <span class="n">DataTypes</span><span class="o">.</span><span class="n">BIGINT</span><span class="p">()))</span> \
+ <span class="o">.</span><span class="n">with_schema</span><span class="p">(</span><span class="n">Schema</span><span class="p">()</span>
+ <span class="o">.</span><span class="n">field</span><span class="p">(</span><span class="s">&#39;sum&#39;</span><span class="p">,</span> <span class="n">DataTypes</span><span class="o">.</span><span class="n">BIGINT</span><span class="p">()))</span> \
+ <span class="o">.</span><span class="n">create_temporary_table</span><span class="p">(</span><span class="s">&#39;mySink&#39;</span><span class="p">)</span>
+
+<span class="n">t_env</span><span class="o">.</span><span class="n">from_path</span><span class="p">(</span><span class="s">&#39;mySource&#39;</span><span class="p">)</span>\
+ <span class="o">.</span><span class="n">select</span><span class="p">(</span><span class="s">&quot;add(a, b)&quot;</span><span class="p">)</span> \
+ <span class="o">.</span><span class="n">insert_into</span><span class="p">(</span><span class="s">&#39;mySink&#39;</span><span class="p">)</span>
+
+<span class="n">t_env</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="s">&quot;tutorial_job&quot;</span><span class="p">)</span></code></pre></div>
+
+<h2 id="submit-the-job">Submit the job</h2>
+
+<p>Firstly, you need to prepare the input data in the “/tmp/input” file. For example,</p>
+
+<p><code>$ echo "1,2" &gt; /tmp/input</code></p>
+
+<p>Next, you can run this example on the command line,</p>
+
+<p><code>$ python python_udf_sum.py</code></p>
+
+<p>The command builds and runs the Python Table API program in a local mini-cluster. You can also submit the Python Table API program to a remote cluster using different command lines, (see more details <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/cli.html#job-submission-examples">here</a>).</p>
+
+<p>Finally, you can see the execution result on the command line:</p>
+
+<p><code>$ cat /tmp/output
+ 3</code></p>
+
+<h2 id="python-udf-dependency-management">Python UDF dependency management</h2>
+
+<p>In many cases, you would like to import third-party dependencies in the Python UDF. The example below provides detailed guidance on how to manage such dependencies.</p>
+
+<p>Suppose you want to use the <code>mpmath</code> to perform the sum of the example above. The Python UDF may look like:</p>
+
+<div class="highlight"><pre><code class="language-python"><span class="nd">@udf</span><span class="p">(</span><span class="n">input_types</span><span class="o">=</span><span class="p">[</span><span class="n">DataTypes</span><span class="o">.</span><span class="n">BIGINT</span>& [...]
+<span class="k">def</span> <span class="nf">add</span><span class="p">(</span><span class="n">i</span><span class="p">,</span> <span class="n">j</span><span class="p">):</span>
+ <span class="kn">from</span> <span class="nn">mpmath</span> <span class="kn">import</span> <span class="n">fadd</span> <span class="c"># add third-party dependency</span>
+ <span class="k">return</span> <span class="nb">int</span><span class="p">(</span><span class="n">fadd</span><span class="p">(</span><span class="n">i</span><span class="p">,</span> <span class="n">j</span><span class="p">))</span></code></pre></div>
+
+<p>To make it available on the worker node that does not contain the dependency, you can specify the dependencies with the following commands and API:</p>
+
+<div class="highlight"><pre><code class="language-bash"><span class="nv">$ </span><span class="nb">cd</span> /tmp
+<span class="nv">$ </span><span class="nb">echo </span><span class="nv">mpmath</span><span class="o">==</span>1.1.0 &gt; requirements.txt
+<span class="nv">$ </span>pip download -d cached_dir -r requirements.txt --no-binary :all:</code></pre></div>
+
+<div class="highlight"><pre><code class="language-python"><span class="n">t_env</span><span class="o">.</span><span class="n">set_python_requirements</span><span class="p">(</span><span class="s">&quot;/tmp/requirements.txt&quot;</span><span class="p">,</span> <span class="s">&quot;/tmp/cached [...]
+
+<p>A <code>requirements.txt</code> file that defines the third-party dependencies is used. If the dependencies cannot be accessed in the cluster, then you can specify a directory containing the installation packages of these dependencies by using the parameter “<code>requirements_cached_dir</code>”, as illustrated in the example above. The dependencies will be uploaded to the cluster and installed offline.</p>
+
+<h1 id="conclusion--upcoming-work">Conclusion &amp; Upcoming work</h1>
+
+<p>In this blog post, we introduced the architecture of Python UDFs in PyFlink and provided some examples on how to define, register and invoke UDFs. Flink 1.10 brings Python support in the framework to new levels, allowing Python users to write even more magic with their preferred language. The community is actively working towards continuously improving the functionality and performance of PyFlink. Future work in upcoming releases will introduce support for Pandas UDFs in scalar [...]
+
+<center>
+<img src="/img/blog/2020-04-09-pyflink-udfs/roadmap-of-pyflink.png" width="600px" alt="Roadmap of PyFlink" />
+</center>
+<p><br /></p>
+</description>
+<pubDate>Thu, 09 Apr 2020 14:00:00 +0200</pubDate>
+<link>https://flink.apache.org/2020/04/09/pyflink-udf-support-flink.html</link>
+<guid isPermaLink="true">/2020/04/09/pyflink-udf-support-flink.html</guid>
+</item>
+
+<item>
+<title>Stateful Functions 2.0 - An Event-driven Database on Apache Flink</title>
+<description><p>Today, we are announcing the release of Stateful Functions (StateFun) 2.0 — the first release of Stateful Functions as part of the Apache Flink project.
+This release marks a big milestone: Stateful Functions 2.0 is not only an API update, but the <strong>first version of an event-driven database</strong> that is built on Apache Flink.</p>
+
+<p>Stateful Functions 2.0 makes it possible to combine StateFun’s powerful approach to state and composition with the elasticity, rapid scaling/scale-to-zero and rolling upgrade capabilities of FaaS implementations like AWS Lambda and modern resource orchestration frameworks like Kubernetes.</p>
+
+<p>With these features, Stateful Functions 2.0 addresses <a href="https://www2.eecs.berkeley.edu/Pubs/TechRpts/2019/EECS-2019-3.pdf">two of the most cited shortcomings</a> of many FaaS setups today: consistent state and efficient messaging between functions.</p>
+
+<div class="page-toc">
+<ul id="markdown-toc">
+ <li><a href="#an-event-driven-database" id="markdown-toc-an-event-driven-database">An Event-driven Database</a></li>
+ <li><a href="#event-driven-database-vs-requestresponse-database" id="markdown-toc-event-driven-database-vs-requestresponse-database">“Event-driven Database” vs. “Request/Response Database”</a> <ul>
+ <li><a href="#state-and-consistency" id="markdown-toc-state-and-consistency">State and Consistency</a></li>
+ </ul>
+ </li>
+ <li><a href="#remote-co-located-or-embedded-functions" id="markdown-toc-remote-co-located-or-embedded-functions">Remote, Co-located or Embedded Functions</a> <ul>
+ <li><a href="#remote-functions" id="markdown-toc-remote-functions">Remote Functions</a></li>
+ <li><a href="#co-located-functions" id="markdown-toc-co-located-functions">Co-located Functions</a></li>
+ <li><a href="#embedded-functions" id="markdown-toc-embedded-functions">Embedded Functions</a></li>
+ </ul>
+ </li>
+ <li><a href="#loading-data-into-the-database" id="markdown-toc-loading-data-into-the-database">Loading Data into the Database</a></li>
+ <li><a href="#try-it-out-and-get-involved" id="markdown-toc-try-it-out-and-get-involved">Try it out and get involved!</a></li>
+ <li><a href="#thank-you" id="markdown-toc-thank-you">Thank you!</a></li>
+</ul>
+
+</div>
+
+<h2 id="an-event-driven-database">An Event-driven Database</h2>
+
+<p>When Stateful Functions joined Apache Flink at the beginning of this year, the project had started as a library on top of Flink to build general-purpose event-driven applications. Users would implement <em>functions</em> that receive and send messages, and maintain state in persistent variables. Flink provided the runtime with efficient exactly-once state and messaging. Stateful Functions 1.0 was a FaaS-inspired mix between stream processing and actor programming — o [...]
+
+<div style="line-height:60%;">
+ <br />
+</div>
+
+<center>
+ <figure>
+ <img src="/img/blog/2020-04-07-release-statefun-2.0.0/image2.png" width="600px" alt="Statefun 1" />
+ <br /><br />
+ <figcaption><i><b>Fig.1:</b> A ride-sharing app as a Stateful Functions example.</i></figcaption>
+ </figure>
+</center>
+
+<div style="line-height:150%;">
+ <br />
+</div>
+
+<p>In version 2.0, Stateful Functions now physically decouples the functions from Flink and the JVM, to invoke them through simple services. That makes it possible to execute functions on a FaaS platform, a Kubernetes deployment or behind a (micro) service.</p>
+
+<p>Flink invokes the functions through a service endpoint via HTTP or gRPC based on incoming events, and supplies state access. The system makes sure that only one invocation per entity (<code>type</code>+<code>ID</code>) is ongoing at any point in time, thus guaranteeing consistency through isolation.
+By supplying state access as part of the function invocation, the functions themselves behave like stateless applications and can be managed with the same simplicity and benefits: rapid scalability, scale-to-zero, rolling/zero-downtime upgrades and so on.</p>
+
+<center>
+ <figure>
+ <img src="/img/blog/2020-04-07-release-statefun-2.0.0/image5.png" width="600px" alt="Statefun 2" />
+ <br /><br />
+ <figcaption><i><b>Fig.2:</b> In Stateful Functions 2.0, functions are stateless and state access is part of the function invocation.</i></figcaption>
+ </figure>
+</center>
+
+<div style="line-height:150%;">
+ <br />
+</div>
+
+<p>The functions can be implemented in any programming language that can handle HTTP requests or bring up a gRPC server. The <a href="https://github.com/apache/flink-statefun">StateFun project</a> includes a very slim SDK for Python, taking requests and dispatching them to annotated functions. We aim to provide similar SDKs for other languages, such as Go, JavaScript or Rust. Users do not need to write any Flink code (or JVM code) at all; data ingresses/egress [...]
+
+<div style="line-height:60%;">
+ <br />
+</div>
+
+<div class="row">
+ <div class="col-lg-6">
+ <div class="text-center">
+ <figure>
+ <img src="/img/blog/2020-04-07-release-statefun-2.0.0/image3.png" width="600px" alt="Statefun 3" />
+ <br /><br />
+ <figcaption><i><b>Fig.3:</b> A module declaring a remote endpoint and a function type.</i></figcaption>
+ </figure>
+ </div>
+ </div>
+ <div class="col-lg-6">
+ <div class="text-center">
+ <figure>
+ <div style="line-height:540%;">
+ <br />
+ </div>
+ <img src="/img/blog/2020-04-07-release-statefun-2.0.0/image10.png" width="600px" alt="Statefun 4" />
+ <br /><br />
+ <figcaption><i><b>Fig.4:</b> A Python implementation of a simple classifier function.</i></figcaption>
+ </figure>
+ </div>
+ </div>
+</div>
+
+<div style="line-height:150%;">
+ <br />
+</div>
+
+<p>The Flink processes (and the JVM) are not executing any user-code at all — though this is possible, for performance reasons (see <a href="#embedded-functions">Embedded Functions</a>). Rather than running application-specific dataflows, Flink here stores the state of the functions and provides the dynamic messaging plane through which functions message each other, carefully dispatching messages/invocations to the event-driven functions/services to maintain c [...]
+
+<blockquote>
+ <p><em>Effectively, Flink takes the role of the database, but tailored towards event-driven functions and services.
+It integrates state storage with the messaging between (and the invocations of) functions and services.
+Because of this, Stateful Functions 2.0 can be thought of as an “Event-driven Database” on Apache Flink.</em></p>
+</blockquote>
+
+<h2 id="event-driven-database-vs-requestresponse-database">“Event-driven Database” vs. “Request/Response Database”</h2>
+
+<p>In the case of a traditional database or key/value store (let’s call them request/response databases), the application issues queries to the database (e.g. SQL via JDBC, GET/PUT via HTTP). In contrast, an event-driven database like StateFun <strong><em>inverts</em></strong> that relationship between database and application: the database invokes the functions/services based on arriving messages. This fits very naturally with FaaS and many event-driven app [...]
+
+<div style="line-height:60%;">
+ <br />
+</div>
+
+<center>
+ <figure>
+ <img src="/img/blog/2020-04-07-release-statefun-2.0.0/image7.png" width="600px" alt="Statefun 5" />
+ <br /><br />
+ <figcaption><i><b>Fig.5:</b> Stateful Functions 2.0 inverts the relationship between database and application.</i></figcaption>
+ </figure>
+</center>
+
+<div style="line-height:150%;">
+ <br />
+</div>
+
+<p>In the case of applications built on request/response databases, the database is responsible only for the state. Communication between different functions/services is a separate concern handled within the application layer. In contrast to that, an event-driven database takes care of both state storage and message transport, in a tightly integrated manner.</p>
+
+<p>Similar to <a href="https://www.brianstorti.com/the-actor-model/">Actor Programming</a>, Stateful Functions uses the idea of <em>addressable entities</em> - here, the entity is a function <code>type</code> with an invocation scoped to an <code>ID</code>. These addressable entities own the state and are the targets of messages. Different to actor systems is that the application logic is external and the addressable entitie [...]
+
+<h3 id="state-and-consistency">State and Consistency</h3>
+
+<p>Besides matching the needs of serverless applications and FaaS well, the event-driven database approach also helps with simplifying consistent state management.</p>
+
+<p>Consider the example below, with two entities of an application — for example two microservices (<em>Service 1</em>, <em>Service 2</em>). <em>Service 1</em> is invoked, updates the state in the database, and sends a request to <em>Service 2</em>. Assume that this request fails. There is, in general, no way for <em>Service 1</em> to know whether <em>Service 2</em> processed the request and updated its state o [...]
+
+<p>In the scenario where the event-driven database takes care of state and messaging, we have a much easier problem to solve. Assume one shard of the database receives the initial message, updates its state, invokes <em>Service 1</em>, and routes the message produced by the function to another shard, to be delivered to <em>Service 2</em>. Now assume message transport errored — it may have failed or not, we cannot know for certain. Because the database is in [...]
+
+<div style="line-height:60%;">
+ <br />
+</div>
+
+<center>
+ <figure>
+ <img src="/img/blog/2020-04-07-release-statefun-2.0.0/image8.png" width="600px" alt="Statefun 6" />
+ <br /><br />
+ <figcaption><i><b>Fig.6:</b> The event-driven database integrates state access and messaging, guaranteeing consistency.</i></figcaption>
+ </figure>
+</center>
+
+<div style="line-height:150%;">
+ <br />
+</div>
+
+<p>That is the big lesson we learned from working on stream processing technology in the past years: <strong>state access/updates and messaging need to be integrated</strong>. This gives you consistency, scalable behavior and backpressures well based on both state access and compute bottlenecks.</p>
+
+<p>Despite state and computation being physically separated here, the scheduling/dispatching of function invocations is still integrated and physically co-located with state access, preserving the consistency guarantees given by physical state/compute co-location.</p>
+
+<h2 id="remote-co-located-or-embedded-functions">Remote, Co-located or Embedded Functions</h2>
+
+<p>Functions can be deployed in various ways that trade off loose coupling and independent scaling with performance overhead. Each module of functions can be of a different kind, so some functions can run remote, while others could run embedded.</p>
+
+<h3 id="remote-functions">Remote Functions</h3>
+
+<p><em>Remote Functions</em> are the mechanism described so far, where functions are deployed separately from the Flink StateFun cluster. The state/messaging tier (i.e. the Flink processes) and the function tier can be deployed and scaled independently. All function invocations are remote and have to go through the endpoint service.</p>
+
+<div style="line-height:60%;">
+ <br />
+</div>
+
+<center>
+<img src="/img/blog/2020-04-07-release-statefun-2.0.0/image6.png" width="600px" alt="Statefun 7" />
+</center>
+
+<div style="line-height:150%;">
+ <br />
+</div>
+
+<p>In a similar way as databases are accessed via a standardized protocol (e.g. ODBC/JDBC for relational databases, REST for many key/value stores), StateFun 2.0 invokes functions and services through a standardized protocol: HTTP or gRPC with data in a well-defined ProtoBuf schema.</p>
+
+<h3 id="co-located-functions">Co-located Functions</h3>
+
+<p>An alternative way of deploying functions is <em>co-location</em> with the Flink JVM processes. In such a setup, each Flink TaskManager would talk to one function process sitting “next to it”. A common way to do this is to use a system like Kubernetes and deploy pods consisting of a Flink container and the function container that communicate via the pod-local network.</p>
+
+<p>This mode supports different languages while avoiding to route invocations through a Service/Gateway/LoadBalancer, but it cannot scale the state and compute parts independently.</p>
+
+<div style="line-height:60%;">
+ <br />
+</div>
+
+<center>
+<img src="/img/blog/2020-04-07-release-statefun-2.0.0/image9.png" width="600px" alt="Statefun 8" />
+</center>
+
+<div style="line-height:150%;">
+ <br />
+</div>
+
+<p>This style of deployment is similar to how <a href="https://beam.apache.org/roadmap/portability/">Apache Beam’s portability layer</a> and <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.11/tutorials/python_table_api.html">Flink’s Python API</a> deploy their non-JVM language SDKs.</p>
+
+<h3 id="embedded-functions">Embedded Functions</h3>
+
+<p><em>Embedded Functions</em> are the mode of Stateful Functions 1.0 and Flink’s Java/Scala stream processing APIs. Functions are deployed into the JVM and are directly invoked with the messages and state access. This is the most performant way, though at the cost of only supporting JVM languages.</p>
+
+<div style="line-height:60%;">
+ <br />
+</div>
+
+<center>
+<img src="/img/blog/2020-04-07-release-statefun-2.0.0/image11.png" width="600px" alt="Statefun 9" />
+</center>
+
+<div style="line-height:150%;">
+ <br />
+</div>
+
+<p>Following the database analogy, embedded functions are a bit like <em>stored procedures</em>, but in a principled way: the functions here are normal Java/Scala/Kotlin functions implementing standard interfaces and can be developed or tested in any IDE.</p>
+
+<h2 id="loading-data-into-the-database">Loading Data into the Database</h2>
+
+<p>When building a new stateful application, you usually don’t start from a completely blank slate. Often, the application has initial state, such as initial “bootstrap” state, or state from previous versions of the application. When using a database, one could simply bulk load the data to prepare the application.</p>
+
+<p>The equivalent step for Flink would be to write a <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/state/savepoints.html">savepoint</a> that contains the initial state. Savepoints are snapshots of the state of the distributed stream processing application and can be passed to Flink to start processing from that state. Think of them as a database dump, but of a distributed streaming database. In the case of StateFun, the savepoint wo [...]
+
+<p>To create a savepoint for a Stateful Functions program, check out the <a href="https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.0/deployment-and-operations/state-bootstrap.html">State Bootstrapping API</a> that is part of StateFun 2.0. The State Bootstrapping API uses Flink’s <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/batch/">DataSet API</a>, but we plan to expand this to use SQL in the [...]
+
+<h2 id="try-it-out-and-get-involved">Try it out and get involved!</h2>
+
+<p>We hope that we could convey some of the excitement we feel about Stateful Functions. If we managed to pique your curiosity, try it out — for example, starting with <a href="https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.0/getting-started/python_walkthrough.html">this walkthrough</a>.</p>
+
+<p>The project is still in a comparatively early stage, so if you want to get involved, there is lots to work on: SDKs for other languages (e.g. Go, JavaScript, Rust), ingresses/egresses and tools for testing, among others.</p>
+
+<p>To follow the project and learn more, please check out these resources:</p>
+
+<ul>
+ <li>Code: <a href="https://github.com/apache/flink-statefun">https://github.com/apache/flink-statefun</a></li>
+ <li>Docs: <a href="https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.0/">https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.0/</a></li>
+ <li>Apache Flink project site: <a href="https://flink.apache.org/">https://flink.apache.org/</a></li>
+ <li>Apache Flink on Twitter: <a href="https://twitter.com/apacheflink">@ApacheFlink</a></li>
+ <li>Stateful Functions Webpage: <a href="https://statefun.io">https://statefun.io</a></li>
+ <li>Stateful Functions on Twitter: <a href="https://twitter.com/statefun_io">@StateFun_IO</a></li>
+</ul>
+
+<h2 id="thank-you">Thank you!</h2>
+
+<p>The Apache Flink community would like to thank all contributors that have made this release possible:</p>
+
+<p>David Anderson, Dian Fu, Igal Shilman, Seth Wiesman, Stephan Ewen, Tzu-Li (Gordon) Tai, hequn8128</p>
+
+</description>
+<pubDate>Tue, 07 Apr 2020 10:00:00 +0200</pubDate>
+<link>https://flink.apache.org/news/2020/04/07/release-statefun-2.0.0.html</link>
+<guid isPermaLink="true">/news/2020/04/07/release-statefun-2.0.0.html</guid>
+</item>
+
+<item>
+<title>Flink Community Update - April'20</title>
+<description><p>While things slow down around us, the Apache Flink community is privileged to remain as active as ever. This blogpost combs through the past few months to give you an update on the state of things in Flink — from core releases to Stateful Functions; from some good old community stats to a new development blog.</p>
+
+<p>And since now it’s more important than ever to keep up the spirits, we’d like to invite you to join the <a href="https://www.flink-forward.org/sf-2020">Flink Forward Virtual Conference</a>, on April 22-24 (see <a href="#upcoming-events">Upcoming Events</a>). Hope to see you there!</p>
+
+<div class="page-toc">
+<ul id="markdown-toc">
+ <li><a href="#the-year-so-far-in-flink" id="markdown-toc-the-year-so-far-in-flink">The Year (so far) in Flink</a> <ul>
+ <li><a href="#flink-110-release" id="markdown-toc-flink-110-release">Flink 1.10 Release</a></li>
+ <li><a href="#stateful-functions-contribution-and-20-release" id="markdown-toc-stateful-functions-contribution-and-20-release">Stateful Functions Contribution and 2.0 Release</a></li>
+ <li><a href="#building-up-to-flink-111" id="markdown-toc-building-up-to-flink-111">Building up to Flink 1.11</a></li>
+ <li><a href="#new-committers-and-pmc-members" id="markdown-toc-new-committers-and-pmc-members">New Committers and PMC Members</a> <ul>
+ <li><a href="#new-pmc-members" id="markdown-toc-new-pmc-members">New PMC Members</a></li>
+ <li><a href="#new-committers" id="markdown-toc-new-committers">New Committers</a></li>
+ </ul>
+ </li>
+ </ul>
+ </li>
+ <li><a href="#the-bigger-picture" id="markdown-toc-the-bigger-picture">The Bigger Picture</a> <ul>
+ <li><a href="#a-look-into-the-flink-repository" id="markdown-toc-a-look-into-the-flink-repository">A Look into the Flink Repository</a></li>
+ <li><a href="#flink-community-packages" id="markdown-toc-flink-community-packages">Flink Community Packages</a></li>
+ <li><a href="#flink-engine-room" id="markdown-toc-flink-engine-room">Flink “Engine Room”</a></li>
+ </ul>
+ </li>
+ <li><a href="#upcoming-events" id="markdown-toc-upcoming-events">Upcoming Events</a> <ul>
+ <li><a href="#flink-forward-virtual-conference" id="markdown-toc-flink-forward-virtual-conference">Flink Forward Virtual Conference</a></li>
+ <li><a href="#others" id="markdown-toc-others">Others</a></li>
+ </ul>
+ </li>
+</ul>
+
+</div>
+
+<h1 id="the-year-so-far-in-flink">The Year (so far) in Flink</h1>
+
+<h2 id="flink-110-release">Flink 1.10 Release</h2>
+
+<p>To kick off the new year, the Flink community <a href="https://flink.apache.org/news/2020/02/11/release-1.10.0.html">released Flink 1.10</a> with the record contribution of over 200 engineers. This release introduced significant improvements to the overall performance and stability of Flink jobs, a preview of native Kubernetes integration and advances in Python support (PyFlink). Flink 1.10 also marked the completion of the <a href="https://flink.ap [...]
+
+<p>The community is now discussing the <a href="http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Releasing-Flink-1-10-1-td38689.html#a38690">release of Flink 1.10.1</a>, covering some outstanding bugs from Flink 1.10.</p>
+
+<h2 id="stateful-functions-contribution-and-20-release">Stateful Functions Contribution and 2.0 Release</h2>
+
+<p>Last January, the first version of Stateful Functions (<a href="https://statefun.io/">statefun.io</a>) code was pushed to the <a href="https://github.com/apache/flink-statefun">Flink repository</a>. Stateful Functions started out as an API to build general purpose event-driven applications on Flink, taking advantage of its advanced state management mechanism to cut the “middleman” that usually handles state coordination in such applica [...]
+
+<p>In a <a href="http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Update-on-Flink-Stateful-Functions-what-are-the-next-steps-tp38646.html">recent update</a>, some new features were announced, like multi-language support (including a Python SDK), function unit testing and Stateful Functions’ own flavor of the <a href="https://ci.apache.org/projects/flink/flink-docs-stable/dev/libs/state_processor_api.html">State Processor A [...]
+
+<h2 id="building-up-to-flink-111">Building up to Flink 1.11</h2>
+
+<p>Amidst the usual outpour of discussion threads, JIRA tickets and FLIPs, the community is working full steam on bringing Flink 1.11 to life in the next few months. The feature freeze is currently scheduled for late April, so the release is expected around mid May.
+The upcoming release will focus on new features and integrations that broaden the scope of Flink use cases, as well as core runtime enhancements to streamline the operations of complex deployments.</p>
+
+<p>Some of the plans on the use case side include support for changelog streams in the Table API/SQL (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-105%3A+Support+to+Interpret+and+Emit+Changelog+in+Flink+SQL">FLIP-105</a>), easy streaming data ingestion into Apache Hive (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-115%3A+Filesystem+connector+in+Table">FLIP-115</a>) and support for Pandas DataFrames in [...]
+
+<p>Throw into the mix improvements around type systems, the WebUI, metrics reporting and supported formats, this release is bound to keep the community busy. For a complete overview of the ongoing development, check <a href="http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Features-of-Apache-Flink-1-11-td38724.html#a38793">this discussion</a> and follow the weekly updates on the Flink <a href="https://flink.apache.org/community.ht [...]
+
+<h2 id="new-committers-and-pmc-members">New Committers and PMC Members</h2>
+
+<p>The Apache Flink community has welcomed <strong>1 PMC (Project Management Committee) Member</strong> and <strong>5 new Committers</strong> since the last update (September 2019):</p>
+
+<h3 id="new-pmc-members">New PMC Members</h3>
+<div class="highlight"><pre><code>Jark Wu
+</code></pre></div>
+
+<h3 id="new-committers">New Committers</h3>
+<div class="highlight"><pre><code>Zili Chen, Jingsong Lee, Yu Li, Dian Fu, Zhu Zhu
+</code></pre></div>
+
+<p>Congratulations to all and thank you for your hardworking commitment to Flink!</p>
+
+<h1 id="the-bigger-picture">The Bigger Picture</h1>
+
+<h2 id="a-look-into-the-flink-repository">A Look into the Flink Repository</h2>
+
+<p>In the <a href="https://flink.apache.org/news/2019/09/10/community-update.html">last update</a>, we shared some numbers around Flink releases and mailing list activity. This time, we’re looking into the activity in the Flink repository and how it’s evolving.</p>
+
+<center>
+<img src="/img/blog/2020-03-30-flink-community-update/2020-03-30-flink-community-update_1.png" width="725px" alt="GitHub 1" />
+</center>
+
+<p>There is a clear upward trend in the number of contributions to the repository, based on the number of commits. This reflects the <strong>fast pace of development</strong> the project is experiencing and also the <strong>successful integration of the China-based Flink contributors</strong> started early last year. To complement these observations, the repository registered a <strong>1.5x increase in the number of individual contributors in 2019</ [...]
+
+<p>But did this increase in capacity produce any other measurable benefits?</p>
+
+<center>
+<img src="/img/blog/2020-03-30-flink-community-update/2020-03-30-flink-community-update_2.png" width="725px" alt="GitHub 2" />
+</center>
+
+<p>If we look at the average time of Pull Request (PR) “resolution”, it seems like it did: <strong>the average time it takes to close a PR these days has been steadily decreasing</strong> since last year, sitting between 5-6 days for the past few months.</p>
+
+<p>These are great indicators of the health of Flink as an open source project!</p>
+
+<h2 id="flink-community-packages">Flink Community Packages</h2>
+
+<p>If you missed the launch of <a href="http://flink-packages.org/">flink-packages.org</a>, here’s a reminder! Ververica has <a href="https://www.ververica.com/blog/announcing-flink-community-packages">created (and open sourced)</a> a website that showcases the work of the community to push forward the ecosystem surrounding Flink. There, you can explore existing packages (like the Pravega and Pulsar Flink connectors, or the Flink Kubernet [...]
+
+<h2 id="flink-engine-room">Flink “Engine Room”</h2>
+
+<p>The community has recently launched the <a href="https://cwiki.apache.org/confluence/pages/viewrecentblogposts.action?key=FLINK">“Engine Room”</a>, a dedicated space in Flink’s Wiki for knowledge sharing between contributors. The goal of this initiative is to make ongoing development on Flink internals more transparent across different work streams, and also to help new contributors get on board with best practices. The first blogpost is already up and shed [...]
+
+<h1 id="upcoming-events">Upcoming Events</h1>
+
+<h2 id="flink-forward-virtual-conference">Flink Forward Virtual Conference</h2>
+
+<p>The organization of Flink Forward had to make the hard decision of cancelling this year’s event in San Francisco. But all is not lost! <strong>Flink Forward SF will be held online on April 22-24 and you can register (for free)</strong> <a href="https://www.flink-forward.org/sf-2020">here</a>. Join the community for interactive talks and Q&amp;A sessions with core Flink contributors and companies like Splunk, Lyft, Netflix or Google.</p>
+
+<h2 id="others">Others</h2>
+
+<p>Events across the globe have come to a halt due to the growing concerns around COVID-19, so this time we’ll leave you with some interesting content to read instead. In addition to this written content, you can also recap last year’s sessions from <a href="https://www.youtube.com/playlist?list=PLDX4T_cnKjD207Aa8b5CsZjc7Z_KRezGz">Flink Forward Berlin</a> and <a href="https://www.youtube.com/playlist?list=PLDX4T_cnKjD3ANoNinSx3Au-poZTHvbF5">Fli [...]
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th>Type</th>
+ <th>Links</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><span class="glyphicon glyphicon glyphicon-bookmark" aria-hidden="true"></span> Blogposts</td>
+ <td><ul>
+ <li><a href="https://medium.com/bird-engineering/replayable-process-functions-in-flink-time-ordering-and-timers-28007a0210e1">Replayable Process Functions: Time, Ordering, and Timers @Bird</a></li>
+ <li><a href="https://engineering.salesforce.com/application-log-intelligence-performance-insights-at-salesforce-using-flink-92955f30573f">Application Log Intelligence &amp; Performance Insights at Salesforce Using Flink @Salesforce</a></li>
+ </ul>
+ <ul>
+ <li><a href="https://flink.apache.org/news/2020/01/29/state-unlocked-interacting-with-state-in-apache-flink.html">State Unlocked: Interacting with State in Apache Flink</a></li>
+ <li><a href="https://flink.apache.org/news/2020/01/15/demo-fraud-detection.html">Advanced Flink Application Patterns Vol.1: Case Study of a Fraud Detection System</a></li>
+ <li><a href="https://flink.apache.org/news/2020/03/24/demo-fraud-detection-2.html">Advanced Flink Application Patterns Vol.2: Dynamic Updates of Application Logic</a></li>
+ <li><a href="https://flink.apache.org/ecosystem/2020/02/22/apache-beam-how-beam-runs-on-top-of-flink.html">Apache Beam: How Beam Runs on Top of Flink</a></li>
+ <li><a href="https://flink.apache.org/features/2020/03/27/flink-for-data-warehouse.html">Flink as Unified Engine for Modern Data Warehousing: Production-Ready Hive Integration</a></li>
+ </ul>
+ </td>
+ </tr>
+ <tr>
+ <td><span class="glyphicon glyphicon-console" aria-hidden="true"></span> Tutorials</td>
+ <td><ul>
+ <li><a href="https://medium.com/@zjffdu/flink-on-zeppelin-part-3-streaming-5fca1e16754">Flink on Zeppelin — (Part 3). Streaming</a></li>
+ <li><a href="https://aws.amazon.com/blogs/big-data/streaming-etl-with-apache-flink-and-amazon-kinesis-data-analytics">Streaming ETL with Apache Flink and Amazon Kinesis Data Analytics</a></li>
+ <li><a href="https://flink.apache.org/news/2020/02/20/ddl.html">No Java Required: Configuring Sources and Sinks in SQL</a></li>
+ <li><a href="https://flink.apache.org/news/2020/02/07/a-guide-for-unit-testing-in-apache-flink.html">A Guide for Unit Testing in Apache Flink</a></li>
+ </ul>
+ </td>
+ </tr>
+ </tbody>
+</table>
+
+<p>If you’d like to keep a closer eye on what’s happening in the community, subscribe to the Flink <a href="https://flink.apache.org/community.html#mailing-lists">@community mailing list</a> to get fine-grained weekly updates, upcoming event announcements and more.</p>
+</description>
+<pubDate>Wed, 01 Apr 2020 10:00:00 +0200</pubDate>
+<link>https://flink.apache.org/news/2020/04/01/community-update.html</link>
+<guid isPermaLink="true">/news/2020/04/01/community-update.html</guid>
+</item>
+
+<item>
+<title>Flink as Unified Engine for Modern Data Warehousing: Production-Ready Hive Integration</title>
+<description><p>In this blog post, you will learn our motivation behind the Flink-Hive integration, and how Flink 1.10 can help modernize your data warehouse.</p>
+
+<div class="page-toc">
+<ul id="markdown-toc">
+ <li><a href="#introduction" id="markdown-toc-introduction">Introduction</a></li>
+ <li><a href="#flink-and-its-integration-with-hive-comes-into-the-scene" id="markdown-toc-flink-and-its-integration-with-hive-comes-into-the-scene">Flink and Its Integration With Hive Comes into the Scene</a> <ul>
+ <li><a href="#unified-metadata-management" id="markdown-toc-unified-metadata-management">Unified Metadata Management</a></li>
+ <li><a href="#stream-processing" id="markdown-toc-stream-processing">Stream Processing</a></li>
+ <li><a href="#compatible-with-more-hive-versions" id="markdown-toc-compatible-with-more-hive-versions">Compatible with More Hive Versions</a></li>
+ <li><a href="#reuse-hive-user-defined-functions-udfs" id="markdown-toc-reuse-hive-user-defined-functions-udfs">Reuse Hive User Defined Functions (UDFs)</a></li>
+ <li><a href="#enhanced-read-and-write-on-hive-data" id="markdown-toc-enhanced-read-and-write-on-hive-data">Enhanced Read and Write on Hive Data</a></li>
+ <li><a href="#formats" id="markdown-toc-formats">Formats</a></li>
+ <li><a href="#more-data-types" id="markdown-toc-more-data-types">More Data Types</a></li>
+ <li><a href="#roadmap" id="markdown-toc-roadmap">Roadmap</a></li>
+ </ul>
+ </li>
+ <li><a href="#summary" id="markdown-toc-summary">Summary</a></li>
+</ul>
+
+</div>
+
+<h2 id="introduction">Introduction</h2>
+
+<p>What are some of the latest requirements for your data warehouse and data infrastructure in 2020?</p>
+
+<p>We’ve came up with some for you.</p>
+
+<p>Firstly, today’s business is shifting to a more real-time fashion, and thus demands abilities to process online streaming data with low latency for near-real-time or even real-time analytics. People become less and less tolerant of delays between when data is generated and when it arrives at their hands, ready to use. Hours or even days of delay is not acceptable anymore. Users are expecting minutes, or even seconds, of end-to-end latency for data in their warehouse, to get quic [...]
+
+<p>Secondly, the infrastructure should be able to handle both offline batch data for offline analytics and exploration, and online streaming data for more timely analytics. Both are indispensable as they both have very valid use cases. Apart from the real time processing mentioned above, batch processing would still exist as it’s good for ad hoc queries and explorations, and full-size calculations. Your modern infrastructure should not force users to choose between one or the other [...]
+
+<p>Thirdly, the data players, including data engineers, data scientists, analysts, and operations, urge a more unified infrastructure than ever before for easier ramp-up and higher working efficiency. The big data landscape has been fragmented for years - companies may have one set of infrastructure for real time processing, one set for batch, one set for OLAP, etc. That, oftentimes, comes as a result of the legacy of lambda architecture, which was popular in the era when stream pr [...]
+
+<p>If any of these resonate with you, you just found the right post to read: we have never been this close to the vision by strengthening Flink’s integration with Hive to a production grade.</p>
+
+<h2 id="flink-and-its-integration-with-hive-comes-into-the-scene">Flink and Its Integration With Hive Comes into the Scene</h2>
+
+<p>Apache Flink has been a proven scalable system to handle extremely high workload of streaming data in super low latency in many giant tech companies.</p>
+
+<p>Despite its huge success in the real time processing domain, at its deep root, Flink has been faithfully following its inborn philosophy of being <a href="https://flink.apache.org/news/2019/02/13/unified-batch-streaming-blink.html">a unified data processing engine for both batch and streaming</a>, and taking a streaming-first approach in its architecture to do batch processing. By making batch a special case for streaming, Flink really leverages its cutting [...]
+
+<p>On the other hand, Apache Hive has established itself as a focal point of the data warehousing ecosystem. It serves as not only a SQL engine for big data analytics and ETL, but also a data management platform, where data is discovered and defined. As business evolves, it puts new requirements on data warehouse.</p>
+
+<p>Thus we started integrating Flink and Hive as a beta version in Flink 1.9. Over the past few months, we have been listening to users’ requests and feedback, extensively enhancing our product, and running rigorous benchmarks (which will be published soon separately). I’m glad to announce that the integration between Flink and Hive is at production grade in <a href="https://flink.apache.org/news/2020/02/11/release-1.10.0.html">Flink 1.10</a> and we can’t wait [...]
+
+<h3 id="unified-metadata-management">Unified Metadata Management</h3>
+
+<p>Hive Metastore has evolved into the de facto metadata hub over the years in the Hadoop, or even the cloud, ecosystem. Many companies have a single Hive Metastore service instance in production to manage all of their schemas, either Hive or non-Hive metadata, as the single source of truth.</p>
+
+<p>In 1.9 we introduced Flink’s <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/hive/hive_catalog.html">HiveCatalog</a>, connecting Flink to users’ rich metadata pool. The meaning of <code>HiveCatalog</code> is two-fold here. First, it allows Apache Flink users to utilize Hive Metastore to store and manage Flink’s metadata, including tables, UDFs, and statistics of data. Second, it enables Flink to access Hive’s exis [...]
+
+<p>In Flink 1.10, users can store Flink’s own tables, views, UDFs, statistics in Hive Metastore on all of the compatible Hive versions mentioned above. <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/hive/hive_catalog.html#example">Here’s an end-to-end example</a> of how to store a Flink’s Kafka source table in Hive Metastore and later query the table in Flink SQL.</p>
+
+<h3 id="stream-processing">Stream Processing</h3>
+
+<p>The Hive integration feature in Flink 1.10 empowers users to re-imagine what they can accomplish with their Hive data and unlock stream processing use cases:</p>
+
+<ul>
+ <li>join real-time streaming data in Flink with offline Hive data for more complex data processing</li>
+ <li>backfill Hive data with Flink directly in a unified fashion</li>
+ <li>leverage Flink to move real-time data into Hive more quickly, greatly shortening the end-to-end latency between when data is generated and when it arrives at your data warehouse for analytics, from hours — or even days — to minutes</li>
+</ul>
+
+<h3 id="compatible-with-more-hive-versions">Compatible with More Hive Versions</h3>
+
+<p>In Flink 1.10, we brought full coverage to most Hive versions including 1.0, 1.1, 1.2, 2.0, 2.1, 2.2, 2.3, and 3.1. Take a look <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/hive/#supported-hive-versions">here</a>.</p>
+
+<h3 id="reuse-hive-user-defined-functions-udfs">Reuse Hive User Defined Functions (UDFs)</h3>
+
+<p>Users can <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/hive/hive_functions.html#hive-user-defined-functions">reuse all kinds of Hive UDFs in Flink</a> since Flink 1.9.</p>
+
+<p>This is a great win for Flink users with past history with the Hive ecosystem, as they may have developed custom business logic in their Hive UDFs. Being able to run these functions without any rewrite saves users a lot of time and brings them a much smoother experience when they migrate to Flink.</p>
+
+<p>To take it a step further, Flink 1.10 introduces <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/hive/hive_functions.html#use-hive-built-in-functions-via-hivemodule">compatibility of Hive built-in functions via HiveModule</a>. Over the years, the Hive community has developed a few hundreds of built-in functions that are super handy for users. For those built-in functions that don’t exist in Flink yet, users are now able to le [...]
+
+<h3 id="enhanced-read-and-write-on-hive-data">Enhanced Read and Write on Hive Data</h3>
+
+<p>Flink 1.10 extends its read and write capabilities on Hive data to all the common use cases with better performance.</p>
+
+<p>On the reading side, Flink now can read Hive regular tables, partitioned tables, and views. Lots of optimization techniques are developed around reading, including partition pruning and projection pushdown to transport less data from file storage, limit pushdown for faster experiment and exploration, and vectorized reader for ORC files.</p>
+
+<p>On the writing side, Flink 1.10 introduces “INSERT INTO” and “INSERT OVERWRITE” to its syntax, and can write to not only Hive’s regular tables, but also partitioned tables with either static or dynamic partitions.</p>
+
+<h3 id="formats">Formats</h3>
+
+<p>Your engine should be able to handle all common types of file formats to give you the freedom of choosing one over another in order to fit your business needs. It’s no exception for Flink. We have tested the following table storage formats: text, csv, SequenceFile, ORC, and Parquet.</p>
+
+<h3 id="more-data-types">More Data Types</h3>
+
+<p>In Flink 1.10, we added support for a few more frequently-used Hive data types that were not covered by Flink 1.9. Flink users now should have a full, smooth experience to query and manipulate Hive data from Flink.</p>
+
+<h3 id="roadmap">Roadmap</h3>
+
+<p>Integration between any two systems is a never-ending story.</p>
+
+<p>We are constantly improving Flink itself and the Flink-Hive integration also gets improved by collecting user feedback and working with folks in this vibrant community.</p>
+
+<p>After careful consideration and prioritization of the feedback we received, we have prioritize many of the below requests for the next Flink release of 1.11.</p>
+
+<ul>
+ <li>Hive streaming sink so that Flink can stream data into Hive tables, bringing a real streaming experience to Hive</li>
+ <li>Native Parquet reader for better performance</li>
+ <li>Additional interoperability - support creating Hive tables, views, functions in Flink</li>
+ <li>Better out-of-box experience with built-in dependencies, including documentations</li>
+ <li>JDBC driver so that users can reuse their existing toolings to run SQL jobs on Flink</li>
+ <li>Hive syntax and semantic compatible mode</li>
+</ul>
+
+<p>If you have more feature requests or discover bugs, please reach out to the community through mailing list and JIRAs.</p>
+
+<h2 id="summary">Summary</h2>
+
+<p>Data warehousing is shifting to a more real-time fashion, and Apache Flink can make a difference for your organization in this space.</p>
+
+<p>Flink 1.10 brings production-ready Hive integration and empowers users to achieve more in both metadata management and unified/batch data processing.</p>
+
+<p>We encourage all our users to get their hands on Flink 1.10. You are very welcome to join the community in development, discussions, and all other kinds of collaborations in this topic.</p>
+
+</description>
+<pubDate>Fri, 27 Mar 2020 03:30:00 +0100</pubDate>
+<link>https://flink.apache.org/features/2020/03/27/flink-for-data-warehouse.html</link>
+<guid isPermaLink="true">/features/2020/03/27/flink-for-data-warehouse.html</guid>
+</item>
+
+<item>
+<title>Advanced Flink Application Patterns Vol.2: Dynamic Updates of Application Logic</title>
+<description><p>In the <a href="https://flink.apache.org/news/2020/01/15/demo-fraud-detection.html">first article</a> of the series, we gave a high-level description of the objectives and required functionality of a Fraud Detection engine. We also described how to make data partitioning in Apache Flink customizable based on modifiable rules instead of using a hardcoded <code>KeysExtractor</code> implementation.</p>
+
+<p>We intentionally omitted details of how the applied rules are initialized and what possibilities exist for updating them at runtime. In this post, we will address exactly these details. You will learn how the approach to data partitioning described in <a href="https://flink.apache.org/news/2020/01/15/demo-fraud-detection.html">Part 1</a> can be applied in combination with a dynamic configuration. These two patterns, when used together, can eliminate the nee [...]
+
+<h2 id="rules-broadcasting">Rules Broadcasting</h2>
+
+<p>Let’s first have a look at the <a href="https://flink.apache.org/news/2020/01/15/demo-fraud-detection.html#dynamic-data-partitioning">previously-defined</a> data-processing pipeline:</p>
+
+<div class="highlight"><pre><code class="language-java"><span class="n">DataStream</span><span class="o">&lt;</span><span class="n">Alert</span><span class="o">&gt;</span> <span class="n">alerts</span> <span class="o">=</span>
+ <span class="n">transactions</span>
+ <span class="o">.</span><span class="na">process</span><span class="o">(</span><span class="k">new</span> <span class="nf">DynamicKeyFunction</span><span class="o">())</span>
+ <span class="o">.</span><span class="na">keyBy</span><span class="o">((</span><span class="n">keyed</span><span class="o">)</span> <span class="o">-&gt;</span> <span class="n">keyed</span><span class="o">.</span><span class="na">getKey</span><span class="o">());< [...]
+ <span class="o">.</span><span class="na">process</span><span class="o">(</span><span class="k">new</span> <span class="nf">DynamicAlertFunction</span><span class="o">())</span></code></pre></div>
+
+<p><code>DynamicKeyFunction</code> provides dynamic data partitioning while <code>DynamicAlertFunction</code> is responsible for executing the main logic of processing transactions and sending alert messages according to defined rules.</p>
+
+<p>Vol.1 of this series simplified the use case and assumed that the applied set of rules is pre-initialized and accessible via the <code>List&lt;Rules&gt;</code> within <code>DynamicKeyFunction</code>.</p>
+
+<div class="highlight"><pre><code class="language-java"><span class="kd">public</span> <span class="kd">class</span> <span class="nc">DynamicKeyFunction</span>
+ <span class="kd">extends</span> <span class="n">ProcessFunction</span><span class="o">&lt;</span><span class="n">Transaction</span><span class="o">,</span> <span class="n">Keyed</span><span class="o">&lt;</span><span class="n">Transaction</span><span class="o">,</span> <span cla [...]
+
+ <span class="cm">/* Simplified */</span>
+ <span class="n">List</span><span class="o">&lt;</span><span class="n">Rule</span><span class="o">&gt;</span> <span class="n">rules</span> <span class="o">=</span> <span class="cm">/* Rules that are initialized somehow.*/</span><span class="o">;</span>
+ <span class="o">...</span>
+<span class="o">}</span></code></pre></div>
+
+<p>Adding rules to this list is obviously possible directly inside the code of the Flink Job at the stage of its initialization (Create a <code>List</code> object; use it’s <code>add</code> method). A major drawback of doing so is that it will require recompilation of the job with each rule modification. In a real Fraud Detection system, rules are expected to change on a frequent basis, making this approach unacceptable from the point of view of business and [...]
+
+<p>Next, let’s take a look at a sample rule definition that we introduced in the previous post of the series:</p>
+
+<center>
+<img src="/img/blog/patterns-blog-2/rule-dsl.png" width="800px" alt="Figure 1: Rule definition" />
+<br />
+<i><small>Figure 1: Rule definition</small></i>
+</center>
+<p><br /></p>
+
+<p>The previous post covered use of <code>groupingKeyNames</code> by <code>DynamicKeyFunction</code> to extract message keys. Parameters from the second part of this rule are used by <code>DynamicAlertFunction</code>: they define the actual logic of the performed operations and their parameters (such as the alert-triggering limit). This means that the same rule must be present in both <code>DynamicKeyFunction</code> and <code>Dy [...]
+
+<p>Figure 2 presents the final job graph of the system that we are building:</p>
+
+<center>
+<img src="/img/blog/patterns-blog-2/job-graph.png" width="800px" alt="Figure 2: Job Graph of the Fraud Detection Flink Job" />
+<br />
+<i><small>Figure 2: Job Graph of the Fraud Detection Flink Job</small></i>
+</center>
+<p><br /></p>
+
+<p>The main blocks of the Transactions processing pipeline are:<br /></p>
+
+<ul>
+ <li>
+ <p><strong>Transaction Source</strong> that consumes transaction messages from Kafka partitions in parallel. <br /></p>
+ </li>
+ <li>
+ <p><strong>Dynamic Key Function</strong> that performs data enrichment with a dynamic key. The subsequent <code>keyBy</code> hashes this dynamic key and partitions the data accordingly among all parallel instances of the following operator.</p>
+ </li>
+ <li>
+ <p><strong>Dynamic Alert Function</strong> that accumulates a data window and creates Alerts based on it.</p>
+ </li>
+</ul>
+
+<h2 id="data-exchange-inside-apache-flink">Data Exchange inside Apache Flink</h2>
+
+<p>The job graph above also indicates various data exchange patterns between the operators. In order to understand how the broadcast pattern works, let’s take a short detour and discuss what methods of message propagation exist in Apache Flink’s distributed runtime.</p>
+
+<ul>
+ <li>The <strong>FORWARD</strong> connection after the Transaction Source means that all data consumed by one of the parallel instances of the Transaction Source operator is transferred to exactly one instance of the subsequent <code>DynamicKeyFunction</code> operator. It also indicates the same level of parallelism of the two connected operators (12 in the above case). This communication pattern is illustrated in Figure 3. Orange circles represent transact [...]
+</ul>
+
+<center>
+<img src="/img/blog/patterns-blog-2/forward.png" width="800px" alt="Figure 3: FORWARD message passing across operator instances" />
+<br />
+<i><small>Figure 3: FORWARD message passing across operator instances</small></i>
+</center>
+<p><br /></p>
+
+<ul>
+ <li>The <strong>HASH</strong> connection between <code>DynamicKeyFunction</code> and <code>DynamicAlertFunction</code> means that for each message a hash code is calculated and messages are evenly distributed among available parallel instances of the next operator. Such a connection needs to be explicitly “requested” from Flink by using <code>keyBy</code>.</li>
+</ul>
+
+<center>
+<img src="/img/blog/patterns-blog-2/hash.png" width="800px" alt="Figure 4: HASHED message passing across operator instances (via `keyBy`)" />
+<br />
+<i><small>Figure 4: HASHED message passing across operator instances (via `keyBy`)</small></i>
+</center>
+<p><br /></p>
+
+<ul>
+ <li>A <strong>REBALANCE</strong> distribution is either caused by an explicit call to <code>rebalance()</code> or by a change of parallelism (12 -&gt; 1 in the case of the job graph from Figure 2). Calling <code>rebalance()</code> causes data to be repartitioned in a round-robin fashion and can help to mitigate data skew in certain scenarios.</li>
+</ul>
+
+<center>
+<img src="/img/blog/patterns-blog-2/rebalance.png" width="800px" alt="Figure 5: REBALANCE message passing across operator instances" />
+<br />
+<i><small>Figure 5: REBALANCE message passing across operator instances</small></i>
+</center>
+<p><br /></p>
+
+<p>The Fraud Detection job graph in Figure 2 contains an additional data source: <em>Rules Source</em>. It also consumes from Kafka. Rules are “mixed into” the main processing data flow through the <strong>BROADCAST</strong> channel. Unlike other methods of transmitting data between operators, such as <code>forward</code>, <code>hash</code> or <code>rebalance</code> that make each message available for processing in only o [...]
+
+<center>
+ <img src="/img/blog/patterns-blog-2/broadcast.png" width="800px" alt="Figure 6: BROADCAST message passing across operator instances" />
+ <br />
+ <i><small>Figure 6: BROADCAST message passing across operator instances</small></i>
+ </center>
+<p><br /></p>
+
+<div class="alert alert-info">
+ <p><span class="label label-info" style="display: inline-block"><span class="glyphicon glyphicon-info-sign" aria-hidden="true"></span> Note</span>
+There are actually a few more specialized data partitioning schemes in Flink which we did not mention here. If you want to find out more, please refer to Flink’s documentation on <strong><a href="https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/#physical-partitioning">stream partitioning</a></strong>.</p>
+</div>
+
+<h2 id="broadcast-state-pattern">Broadcast State Pattern</h2>
+
+<p>In order to make use of the Rules Source, we need to “connect” it to the main data stream:</p>
+
+<div class="highlight"><pre><code class="language-java"><span class="c1">// Streams setup</span>
+<span class="n">DataStream</span><span class="o">&lt;</span><span class="n">Transaction</span><span class="o">&gt;</span> <span class="n">transactions</span> <span class="o">=</span> <span class="o">[...]</span>
+<span class="n">DataStream</span><span class="o">&lt;</span><span class="n">Rule</span><span class="o">&gt;</span> <span class="n">rulesUpdateStream</span> <span class="o">=</span> <span class="o">[...]</span>
+
+<span class="n">BroadcastStream</span><span class="o">&lt;</span><span class="n">Rule</span><span class="o">&gt;</span> <span class="n">rulesStream</span> <span class="o">=</span> <span class="n">rulesUpdateStream</span><span class="o">.</span><span class="na">broadcast</span><span [...]
+
+<span class="c1">// Processing pipeline setup</span>
+ <span class="n">DataStream</span><span class="o">&lt;</span><span class="n">Alert</span><span class="o">&gt;</span> <span class="n">alerts</span> <span class="o">=</span>
+ <span class="n">transactions</span>
+ <span class="o">.</span><span class="na">connect</span><span class="o">(</span><span class="n">rulesStream</span><span class="o">)</span>
+ <span class="o">.</span><span class="na">process</span><span class="o">(</span><span class="k">new</span> <span class="nf">DynamicKeyFunction</span><span class="o">())</span>
+ <span class="o">.</span><span class="na">keyBy</span><span class="o">((</span><span class="n">keyed</span><span class="o">)</span> <span class="o">-&gt;</span> <span class="n">keyed</span><span class="o">.</span><span class="na">getKey</span><span class="o">())< [...]
+ <span class="o">.</span><span class="na">connect</span><span class="o">(</span><span class="n">rulesStream</span><span class="o">)</span>
+ <span class="o">.</span><span class="na">process</span><span class="o">(</span><span class="k">new</span> <span class="nf">DynamicAlertFunction</span><span class="o">())</span></code></pre></div>
+
+<p>As you can see, the broadcast stream can be created from any regular stream by calling the <code>broadcast</code> method and specifying a state descriptor. Flink assumes that broadcasted data needs to be stored and retrieved while processing events of the main data flow and, therefore, always automatically creates a corresponding <em>broadcast state</em> from this state descriptor. This is different from any other Apache Flink state type in which you need [...]
+
+<div class="highlight"><pre><code class="language-java"><span class="kd">public</span> <span class="kd">static</span> <span class="kd">final</span> <span class="n">MapStateDescriptor</span><span class="o">&lt;</span><span class="n">Integer</span><span class="o">,</span> <span class="n&qu [...]
+ <span class="k">new</span> <span class="n">MapStateDescriptor</span><span class="o">&lt;&gt;(</span><span class="s">&quot;rules&quot;</span><span class="o">,</span> <span class="n">Integer</span><span class="o">.</span><span class="na">class</span><span class="o">,</span&g [...]
+
+<p>Connecting to <code>rulesStream</code> causes some changes in the signature of the processing functions. The previous article presented it in a slightly simplified way as a <code>ProcessFunction</code>. However, <code>DynamicKeyFunction</code> is actually a <code>BroadcastProcessFunction</code>.</p>
+
+<div class="highlight"><pre><code class="language-java"><span class="kd">public</span> <span class="kd">abstract</span> <span class="kd">class</span> <span class="nc">BroadcastProcessFunction</span><span class="o">&lt;</span><span class="n">IN1</span><span class="o">,</span> <span class=" [...]
+
+ <span class="kd">public</span> <span class="kd">abstract</span> <span class="kt">void</span> <span class="nf">processElement</span><span class="o">(</span><span class="n">IN1</span> <span class="n">value</span><span class="o">,</span>
+ <span class="n">ReadOnlyContext</span> <span class="n">ctx</span><span class="o">,</span>
+ <span class="n">Collector</span><span class="o">&lt;</span><span class="n">OUT</span><span class="o">&gt;</span> <span class="n">out</span><span class="o">)</span> <span class="kd">throws</span> <span class="n">Exception</span><span class="o">;</span>
+
+ <span class="kd">public</span> <span class="kd">abstract</span> <span class="kt">void</span> <span class="nf">processBroadcastElement</span><span class="o">(</span><span class="n">IN2</span> <span class="n">value</span><span class="o">,</span>
+ <span class="n">Context</span> <span class="n">ctx</span><span class="o">,</span>
+ <span class="n">Collector</span><span class="o">&lt;</span><span class="n">OUT</span><span class="o">&gt;</span> <span class="n">out</span><span class="o">)</span> <span class="kd">throws</span> <span class="n">Exception</span><span class="o"> [...]
+
+<span class="o">}</span></code></pre></div>
+
+<p>The difference is the addition of the <code>processBroadcastElement</code> method through which messages of the rules stream will arrive. The following new version of <code>DynamicKeyFunction</code> allows modifying the list of data-distribution keys at runtime through this stream:</p>
+
+<div class="highlight"><pre><code class="language-java"><span class="kd">public</span> <span class="kd">class</span> <span class="nc">DynamicKeyFunction</span>
+ <span class="kd">extends</span> <span class="n">BroadcastProcessFunction</span><span class="o">&lt;</span><span class="n">Transaction</span><span class="o">,</span> <span class="n">Rule</span><span class="o">,</span> <span class="n">Keyed</span><span class="o">&lt;</span><span c [...]
+
+
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="nf">processBroadcastElement</span><span class="o">(</span><span class="n">Rule</span> <span class="n">rule</span><span class="o">,</span>
+ <span class="n">Context</span> <span class="n">ctx</span><span class="o">,</span>
+ <span class="n">Collector</span><span class="o">&lt;</span><span class="n">Keyed</span><span class="o">&lt;</span><span class="n">Transaction</span><span class="o">,</span> <span class="n">String</span><span class="o">,</span> <span class="n">Integer</ [...]
+ <span class="n">BroadcastState</span><span class="o">&lt;</span><span class="n">Integer</span><span class="o">,</span> <span class="n">Rule</span><span class="o">&gt;</span> <span class="n">broadcastState</span> <span class="o">=</span> <span class="n">ctx</span><span class=&quo [...]
+ <span class="n">broadcastState</span><span class="o">.</span><span class="na">put</span><span class="o">(</span><span class="n">rule</span><span class="o">.</span><span class="na">getRuleId</span><span class="o">(),</span> <span class="n">rule</span><span class="o">);</span>
+ <span class="o">}</span>
+
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="nf">processElement</span><span class="o">(</span><span class="n">Transaction</span> <span class="n">event</span><span class="o">,</span>
+ <span class="n">ReadOnlyContext</span> <span class="n">ctx</span><span class="o">,</span>
+ <span class="n">Collector</span><span class="o">&lt;</span><span class="n">Keyed</span><span class="o">&lt;</span><span class="n">Transaction</span><span class="o">,</span> <span class="n">String</span><span class="o">,</span> <span class="n">Integer</span>&l [...]
+ <span class="n">ReadOnlyBroadcastState</span><span class="o">&lt;</span><span class="n">Integer</span><span class="o">,</span> <span class="n">Rule</span><span class="o">&gt;</span> <span class="n">rulesState</span> <span class="o">=</span>
+ <span class="n">ctx</span><span class="o">.</span><span class="na">getBroadcastState</span><span class="o">(</span><span class="n">RULES_STATE_DESCRIPTOR</span><span class="o">);</span>
+ <span class="k">for</span> <span class="o">(</span><span class="n">Map</span><span class="o">.</span><span class="na">Entry</span><span class="o">&lt;</span><span class="n">Integer</span><span class="o">,</span> <span class="n">Rule</span><span class="o">&gt;</sp [...]
+ <span class="kd">final</span> <span class="n">Rule</span> <span class="n">rule</span> <span class="o">=</span> <span class="n">entry</span><span class="o">.</span><span class="na">getValue</span><span class="o">();</span>
+ <span class="n">out</span><span class="o">.</span><span class="na">collect</span><span class="o">(</span>
+ <span class="k">new</span> <span class="n">Keyed</span><span class="o">&lt;&gt;(</span>
+ <span class="n">event</span><span class="o">,</span> <span class="n">KeysExtractor</span><span class="o">.</span><span class="na">getKey</span><span class="o">(</span><span class="n">rule</span><span class="o">.</span><span class="na">getGroupingKeyNames</span><span class=" [...]
+ <span class="o">}</span>
+ <span class="o">}</span>
+<span class="o">}</span></code></pre></div>
+
+<p>In the above code, <code>processElement()</code> receives Transactions, and <code>processBroadcastElement()</code> receives Rule updates. When a new rule is created, it is distributed as depicted in Figure 6 and saved in all parallel instances of the operator using <code>processBroadcastState</code>. We use a Rule’s ID as the key to store and reference individual rules. Instead of iterating over a hardcoded <code>List&lt;Rules&gt [...]
+
+<p><code>DynamicAlertFunction</code> follows the same logic with respect to storing the rules in the broadcast <code>MapState</code>. As described in <a href="https://flink.apache.org/news/2020/01/15/demo-fraud-detection.html">Part 1</a>, each message in the <code>processElement</code> input is intended to be processed by one specific rule and comes “pre-marked” with a corresponding ID by <code>DynamicKeyFunction</ [...]
+
+<h1 id="summary">Summary</h1>
+
+<p>In this blog post, we continued our investigation of the use case of a Fraud Detection System built with Apache Flink. We looked into different ways in which data can be distributed between parallel operator instances and, most importantly, examined broadcast state. We demonstrated how dynamic partitioning — a pattern described in the <a href="https://flink.apache.org/news/2020/01/15/demo-fraud-detection.html">first part</a> of the series — can be combined [...]
+</description>
+<pubDate>Tue, 24 Mar 2020 13:00:00 +0100</pubDate>
+<link>https://flink.apache.org/news/2020/03/24/demo-fraud-detection-2.html</link>
+<guid isPermaLink="true">/news/2020/03/24/demo-fraud-detection-2.html</guid>
+</item>
+
+<item>
+<title>Apache Beam: How Beam Runs on Top of Flink</title>
+<description><p>Note: This blog post is based on the talk <a href="https://www.youtube.com/watch?v=hxHGLrshnCY">“Beam on Flink: How Does It Actually Work?”</a>.</p>
+
+<p><a href="https://flink.apache.org/">Apache Flink</a> and <a href="https://beam.apache.org/">Apache Beam</a> are open-source frameworks for parallel, distributed data processing at scale. Unlike Flink, Beam does not come with a full-blown execution engine of its own but plugs into other execution engines, such as Apache Flink, Apache Spark, or Google Cloud Dataflow. In this blog post we discuss the reasons to use Flink together with Bea [...]
+
+<h1 id="what-is-apache-beam">What is Apache Beam</h1>
+
+<p><a href="https://beam.apache.org/">Apache Beam</a> is an open-source, unified model for defining batch and streaming data-parallel processing pipelines. It is unified in the sense that you use a single API, in contrast to using a separate API for batch and streaming like it is the case in Flink. Beam was originally developed by Google which released it in 2014 as the Cloud Dataflow SDK. In 2016, it was donated to <a href="https://www.apache.org/&quo [...]
+
+<p>The execution model, as well as the API of Apache Beam, are similar to Flink’s. Both frameworks are inspired by the <a href="https://static.googleusercontent.com/media/research.google.com/en//archive/mapreduce-osdi04.pdf">MapReduce</a>, <a href="https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/41378.pdf">MillWheel</a>, and <a href="https://research.google/pubs/pub43864/">Dataflow</a> [...]
+
+<p>One of the most exciting developments in the Beam technology is the framework’s support for multiple programming languages including Java, Python, Go, Scala and SQL. Essentially, developers can write their applications in a programming language of their choice. Beam, with the help of the Runners, translates the program to one of the execution engines, as shown in the diagram below.</p>
+
+<center>
+<img src="/img/blog/2020-02-22-beam-on-flink/flink-runner-beam-beam-vision.png" width="600px" alt="The vision of Apache Beam" />
+</center>
+
+<h1 id="reasons-to-use-beam-with-flink">Reasons to use Beam with Flink</h1>
+
+<p>Why would you want to use Beam with Flink instead of directly using Flink? Ultimately, Beam and Flink complement each other and provide additional value to the user. The main reasons for using Beam with Flink are the following:</p>
+
+<ul>
+ <li>Beam provides a unified API for both batch and streaming scenarios.</li>
+ <li>Beam comes with native support for different programming languages, like Python or Go with all their libraries like Numpy, Pandas, Tensorflow, or TFX.</li>
+ <li>You get the power of Apache Flink like its exactly-once semantics, strong memory management and robustness.</li>
+ <li>Beam programs run on your existing Flink infrastructure or infrastructure for other supported Runners, like Spark or Google Cloud Dataflow.</li>
+ <li>You get additional features like side inputs and cross-language pipelines that are not supported natively in Flink but only supported when using Beam with Flink.</li>
+</ul>
+
+<h1 id="the-flink-runner-in-beam">The Flink Runner in Beam</h1>
+
+<p>The Flink Runner in Beam translates Beam pipelines into Flink jobs. The translation can be parameterized using Beam’s pipeline options which are parameters for settings like configuring the job name, parallelism, checkpointing, or metrics reporting.</p>
+
+<p>If you are familiar with a DataSet or a DataStream, you will have no problems understanding what a PCollection is. PCollection stands for parallel collection in Beam and is exactly what DataSet/DataStream would be in Flink. Due to Beam’s unified API we only have one type of results of transformation: PCollection.</p>
+
+<p>Beam pipelines are composed of transforms. Transforms are like operators in Flink and come in two flavors: primitive and composite transforms. The beauty of all this is that Beam only comes with a small set of primitive transforms which are:</p>
+
+<ul>
+ <li><code>Source</code> (for loading data)</li>
+ <li><code>ParDo</code> (think of a flat map operator on steroids)</li>
+ <li><code>GroupByKey</code> (think of keyBy() in Flink)</li>
+ <li><code>AssignWindows</code> (windows can be assigned at any point in time in Beam)</li>
+ <li><code>Flatten</code> (like a union() operation in Flink)</li>
+</ul>
+
+<p>Composite transforms are built by combining the above primitive transforms. For example, <code>Combine = GroupByKey + ParDo</code>.</p>
+
+<h1 id="flink-runner-internals">Flink Runner Internals</h1>
+
+<p>Although using the Flink Runner in Beam has no prerequisite to understanding its internals, we provide more details of how the Flink runner works in Beam to share knowledge of how the two frameworks can integrate and work together to provide state-of-the-art streaming data pipelines.</p>
+
+<p>The Flink Runner has two translation paths. Depending on whether we execute in batch or streaming mode, the Runner either translates into Flink’s DataSet or into Flink’s DataStream API. Since multi-language support has been added to Beam, another two translation paths have been added. To summarize the four modes:</p>
+
+<ol>
+ <li><strong>The Classic Flink Runner for batch jobs:</strong> Executes batch Java pipelines</li>
+ <li><strong>The Classic Flink Runner for streaming jobs:</strong> Executes streaming Java pipelines</li>
+ <li><strong>The Portable Flink Runner for batch jobs:</strong> Executes Java as well as Python, Go and other supported SDK pipelines for batch scenarios</li>
+ <li><strong>The Portable Flink Runner for streaming jobs:</strong> Executes Java as well as Python, Go and other supported SDK pipelines for streaming scenarios</li>
+</ol>
+
+<center>
+<img src="/img/blog/2020-02-22-beam-on-flink/flink-runner-beam-runner-translation-paths.png" width="300px" alt="The 4 translation paths in the Beam's Flink Runner" />
+</center>
+
+<h2 id="the-classic-flink-runner-in-beam">The “Classic” Flink Runner in Beam</h2>
+
+<p>The classic Flink Runner was the initial version of the Runner, hence the “classic” name. Beam pipelines are represented as a graph in Java which is composed of the aforementioned composite and primitive transforms. Beam provides translators which traverse the graph in topological order. Topological order means that we start from all the sources first as we iterate through the graph. Presented with a transform from the graph, the Flink Runner generates the API calls as you would [...]
+
+<center>
+<img src="/img/blog/2020-02-22-beam-on-flink/classic-flink-runner-beam.png" width="600px" alt="The Classic Flink Runner in Beam" />
+</center>
+
+<p>While Beam and Flink share very similar concepts, there are enough differences between the two frameworks that make Beam pipelines impossible to be translated 1:1 into a Flink program. In the following sections, we will present the key differences:</p>
+
+<h3 id="serializers-vs-coders">Serializers vs Coders</h3>
+
+<p>When data is transferred over the wire in Flink, it has to be turned into bytes. This is done with the help of serializers. Flink has a type system to instantiate the correct coder for a given type, e.g. <code>StringTypeSerializer</code> for a String. Apache Beam also has its own type system which is similar to Flink’s but uses slightly different interfaces. Serializers are called Coders in Beam. In order to make a Beam Coder run in Flink, we have to make the two ser [...]
+
+<center>
+<img src="/img/blog/2020-02-22-beam-on-flink/flink-runner-beam-serializers-coders.png" width="300px" alt="Serializers vs Coders" />
+</center>
+
+<h3 id="read">Read</h3>
+
+<p>The <code>Read</code> transform provides a way to read data into your pipeline in Beam. The Read transform is supported by two wrappers in Beam, the <code>SourceInputFormat</code> for batch processing and the <code>UnboundedSourceWrapper</code> for stream processing.</p>
+
+<h3 id="pardo">ParDo</h3>
+
+<p><code>ParDo</code> is the swiss army knife of Beam and can be compared to a <code>RichFlatMapFunction</code> in Flink with additional features such as <code>SideInputs</code>, <code>SideOutputs</code>, State and Timers. <code>ParDo</code> is essentially translated by the Flink runner using the <code>FlinkDoFnFunction</code> for batch processing or the <code>FlinkStatefulDoFnFunction</code>, while [...]
+
+<h3 id="side-inputs">Side Inputs</h3>
+
+<p>In addition to the main input, ParDo transforms can have a number of side inputs. A side input can be a static set of data that you want to have available at all parallel instances. However, it is more flexible than that. You can have keyed and even windowed side input which updates based on the window size. This is a very powerful concept which does not exist in Flink but is added on top of Flink using Beam.</p>
+
+<h3 id="assignwindows">AssignWindows</h3>
+
+<p>In Flink, windows are assigned by the <code>WindowOperator</code> when you use the <code>window()</code> in the API. In Beam, windows can be assigned at any point in time. Any element is implicitly part of a window. If no window is assigned explicitly, the element is part of the <code>GlobalWindow</code>. Window information is stored for each element in a wrapper called <code>WindowedValue</code>. The window information is only use [...]
+
+<h3 id="groupbykey">GroupByKey</h3>
+
+<p>Most of the time it is useful to partition the data by a key. In Flink, this is done via the <code>keyBy()</code> API call. In Beam the <code>GroupByKey</code> transform can only be applied if the input is of the form <code>KV&lt;Key, Value&gt;</code>. Unlike Flink where the key can even be nested inside the data, Beam enforces the key to always be explicit. The <code>GroupByKey</code> transform then groups the data by key [...]
+
+<h3 id="flatten">Flatten</h3>
+
+<p>The Flatten operator takes multiple DataSet/DataStreams, called P[arallel]Collections in Beam, and combines them into one collection. This is equivalent to Flink’s <code>union()</code> operation.</p>
+
+<h2 id="the-portable-flink-runner-in-beam">The “Portable” Flink Runner in Beam</h2>
+
+<p>The portable Flink Runner in Beam is the evolution of the classic Runner. Classic Runners are tied to the JVM ecosystem, but the Beam community wanted to move past this and also execute Python, Go and other languages. This adds another dimension to Beam in terms of portability because, like previously mentioned, Beam already had portability across execution engines. It was necessary to change the translation logic of the Runner to be able to support language portability.</p>
+
+<p>There are two important building blocks for portable Runners:</p>
+
+<ol>
+ <li>A common pipeline format across all the languages: The Runner API</li>
+ <li>A common interface during execution for the communication between the Runner and the code written in any language: The Fn API</li>
+</ol>
+
+<p>The Runner API provides a universal representation of the pipeline as Protobuf which contains the transforms, types, and user code. Protobuf was chosen as the format because every language has libraries available for it. Similarly, for the execution part, Beam introduced the Fn API interface to handle the communication between the Runner/execution engine and the user code that may be written in a different language and executes in a different process. Fn API is pronounced “fun A [...]
+
+<center>
+<img src="/img/blog/2020-02-22-beam-on-flink/flink-runner-beam-language-portability.png" width="600px" alt="Language Portability in Apache Beam" />
+</center>
+
+<h2 id="how-are-beam-programs-translated-in-language-portability">How Are Beam Programs Translated In Language Portability?</h2>
+
+<p>Users write their Beam pipelines in one language, but they may get executed in an environment based on a completely different language. How does that work? To explain that, let’s follow the lifecycle of a pipeline. Let’s suppose we use the Python SDK to write the pipeline. Before submitting the pipeline via the Job API to Beam’s JobServer, Beam would convert it to the Runner API, the language-agnostic format we described before. The JobServer is also a Beam component that handle [...]
+
+<ul>
+ <li>Docker-based (the default)</li>
+ <li>Process-based (a simple process is started)</li>
+ <li>Externally-provided (K8s or other schedulers)</li>
+ <li>Embedded (intended for testing and only works with Java)</li>
+</ul>
+
+<p>Environments hold the <em>SDK Harness</em> which is the code that handles the execution and the communication with the Runner over the Fn API. For example, when Flink executes Python code, it sends the data to the Python environment containing the Python SDK Harness. Sending data to an external process involves a minor overhead which we have measured to be 5-10% slower than the classic Java pipelines. However, Beam uses a fusion of transforms to execute as many trans [...]
+
+<center>
+<img src="/img/blog/2020-02-22-beam-on-flink/flink-runner-beam-language-portability-architecture.png" width="600px" alt="Language Portability Architecture in beam" />
+</center>
+
+<p>Environments can be present for many languages. This opens up an entirely new type of pipelines: cross-language pipelines. In cross-language pipelines we can combine transforms of two or more languages, e.g. a machine learning pipeline with the feature generation written in Java and the learning written in Python. All this can be run on top of Flink.</p>
+
+<h2 id="conclusion">Conclusion</h2>
+
+<p>Using Apache Beam with Apache Flink combines (a.) the power of Flink with (b.) the flexibility of Beam. All it takes to run Beam is a Flink cluster, which you may already have. Apache Beam’s fully-fledged Python API is probably the most compelling argument for using Beam with Flink, but the unified API which allows to “write-once” and “execute-anywhere” is also very appealing to Beam users. On top of this, features like side inputs and a rich connector ecosystem are also reason [...]
+
+<p>With the introduction of schemas, a new format for handling type information, Beam is heading in a similar direction as Flink with its type system which is essential for the Table API or SQL. Speaking of, the next Flink release will include a Python version of the Table API which is based on the language portability of Beam. Looking ahead, the Beam community plans to extend the support for interactive programs like notebooks. TFX, which is built with Beam, is a very powerful way [...]
+
+<p>For many years, Beam and Flink have inspired and learned from each other. With the Python support being based on Beam in Flink, they only seem to come closer to each other. That’s all the better for the community, and also users have more options and functionality to choose from.</p>
+</description>
+<pubDate>Sat, 22 Feb 2020 13:00:00 +0100</pubDate>
+<link>https://flink.apache.org/ecosystem/2020/02/22/apache-beam-how-beam-runs-on-top-of-flink.html</link>
+<guid isPermaLink="true">/ecosystem/2020/02/22/apache-beam-how-beam-runs-on-top-of-flink.html</guid>
+</item>
+
+<item>
+<title>No Java Required: Configuring Sources and Sinks in SQL</title>
+<description><h1 id="introduction">Introduction</h1>
+
+<p>The recent <a href="https://flink.apache.org/news/2020/02/11/release-1.10.0.html">Apache Flink 1.10 release</a> includes many exciting features.
+In particular, it marks the end of the community’s year-long effort to merge in the <a href="https://flink.apache.org/news/2019/02/13/unified-batch-streaming-blink.html">Blink SQL contribution</a> from Alibaba.
+The reason the community chose to spend so much time on the contribution is that SQL works.
+It allows Flink to offer a truly unified interface over batch and streaming and makes stream processing accessible to a broad audience of developers and analysts.
+Best of all, Flink SQL is ANSI-SQL compliant, which means if you’ve ever used a database in the past, you already know it<sup id="fnref:1"><a href="#fn:1" class="footnote">1</a></sup>!</p>
+
+<p>A lot of work focused on improving runtime performance and progressively extending its coverage of the SQL standard.
+Flink now supports the full TPC-DS query set for batch queries, reflecting the readiness of its SQL engine to address the needs of modern data warehouse-like workloads.
+Its streaming SQL supports an almost equal set of features - those that are well defined on a streaming runtime - including <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/streaming/joins.html">complex joins</a> and <a href="https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/streaming/match_recognize.html">MATCH_RECOGNIZE</a>.</p>
+
+<p>As important as this work is, the community also strives to make these features generally accessible to the broadest audience possible.
+That is why the Flink community is excited in 1.10 to offer production-ready DDL syntax (e.g., <code>CREATE TABLE</code>, <code>DROP TABLE</code>) and a refactored catalog interface.</p>
+
+<h1 id="accessing-your-data-where-it-lives">Accessing Your Data Where It Lives</h1>
+
+<p>Flink does not store data at rest; it is a compute engine and requires other systems to consume input from and write its output.
+Those that have used Flink’s <code>DataStream</code> API in the past will be familiar with connectors that allow for interacting with external systems.
+Flink has a vast connector ecosystem that includes all major message queues, filesystems, and databases.</p>
+
+<div class="alert alert-info">
+If your favorite system does not have a connector maintained in the central Apache Flink repository, check out the <a href="https://flink-packages.org">flink packages website</a>, which has a growing number of community-maintained components.
+</div>
+
+<p>While these connectors are battle-tested and production-ready, they are written in Java and configured in code, which means they are not amenable to pure SQL or Table applications.
+For a holistic SQL experience, not only queries need to be written in SQL, but also table definitions.</p>
+
+<h1 id="create-table-statements">CREATE TABLE Statements</h1>
+
+<p>While Flink SQL has long provided table abstractions atop some of Flink’s most popular connectors, configurations were not always so straightforward.
+Beginning in 1.10, Flink supports defining tables through <code>CREATE TABLE</code> statements.
+With this feature, users can now create logical tables, backed by various external systems, in pure SQL.</p>
+
+<p>By defining tables in SQL, developers can write queries against logical schemas that are abstracted away from the underlying physical data store. Coupled with Flink SQL’s unified approach to batch and stream processing, Flink provides a straight line from discovery to production.</p>
+
+<p>Users can define tables over static data sets, anything from a local CSV file to a full-fledged data lake or even Hive.
+Leveraging Flink’s efficient batch processing capabilities, they can perform ad-hoc queries searching for exciting insights.
+Once something interesting is identified, businesses can gain real-time and continuous insights by merely altering the table so that it is powered by a message queue such as Kafka.
+Because Flink guarantees SQL queries have unified semantics over batch and streaming, users can be confident that redeploying this query as a continuous streaming application over a message queue will output identical results.</p>
+
+<figure class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="c1">-- Define a table called orders that is backed by a Kafka topic</span>
+<span class="c1">-- The definition includes all relevant Kafka properties,</span>
+<span class="c1">-- the underlying format (JSON) and even defines a</span>
+<span class="c1">-- watermarking algorithm based on one of the fields</span>
+<span class="c1">-- so that this table can be used with event time.</span>
+<span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">orders</span> <span class="p">(</span>
+ <span class="n">user_id</span> <span class="nb">BIGINT</span><span class="p">,</span>
+ <span class="n">product</span> <span class="n">STRING</span><span class="p">,</span>
+ <span class="n">order_time</span> <span class="k">TIMESTAMP</span><span class="p">(</span><span class="mi">3</span><span class="p">),</span>
+ <span class="n">WATERMARK</span> <span class="k">FOR</span> <span class="n">order_time</span> <span class="k">AS</span> <span class="n">order_time</span> <span class="o">-</span> <span class="s1">&#39;5&#39;</span> <span class="n">SECONDS</span>
+<span class="p">)</span> <span class="k">WITH</span> <span class="p">(</span>
+ <span class="s1">&#39;connector.type&#39;</span> <span class="o">=</span> <span class="s1">&#39;kafka&#39;</span><span class="p">,</span>
+ <span class="s1">&#39;connector.version&#39;</span> <span class="o">=</span> <span class="s1">&#39;universal&#39;</span><span class="p">,</span>
+ <span class="s1">&#39;connector.topic&#39;</span> <span class="o">=</span> <span class="s1">&#39;orders&#39;</span><span class="p">,</span>
+ <span class="s1">&#39;connector.startup-mode&#39;</span> <span class="o">=</span> <span class="s1">&#39;earliest-offset&#39;</span><span class="p">,</span>
+ <span class="s1">&#39;connector.properties.bootstrap.servers&#39;</span> <span class="o">=</span> <span class="s1">&#39;localhost:9092&#39;</span><span class="p">,</span>
+ <span class="s1">&#39;format.type&#39;</span> <span class="o">=</span> <span class="s1">&#39;json&#39;</span>
+<span class="p">);</span>
+
+<span class="c1">-- Define a table called product_analysis</span>
+<span class="c1">-- on top of ElasticSearch 7 where we </span>
+<span class="c1">-- can write the results of our query. </span>
+<span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">product_analysis</span> <span class="p">(</span>
+ <span class="n">product</span> <span class="n">STRING</span><span class="p">,</span>
+ <span class="n">tracking_time</span> <span class="k">TIMESTAMP</span><span class="p">(</span><span class="mi">3</span><span class="p">),</span>
+ <span class="n">units_sold</span> <span class="nb">BIGINT</span>
+<span class="p">)</span> <span class="k">WITH</span> <span class="p">(</span>
+ <span class="s1">&#39;connector.type&#39;</span> <span class="o">=</span> <span class="s1">&#39;elasticsearch&#39;</span><span class="p">,</span>
+ <span class="s1">&#39;connector.version&#39;</span> <span class="o">=</span> <span class="s1">&#39;7&#39;</span><span class="p">,</span>
+ <span class="s1">&#39;connector.hosts&#39;</span> <span class="o">=</span> <span class="s1">&#39;localhost:9200&#39;</span><span class="p">,</span>
+ <span class="s1">&#39;connector.index&#39;</span> <span class="o">=</span> <span class="s1">&#39;ProductAnalysis&#39;</span><span class="p">,</span>
+ <span class="s1">&#39;connector.document.type&#39;</span> <span class="o">=</span> <span class="s1">&#39;analysis&#39;</span>
+<span class="p">);</span>
+
+<span class="c1">-- A simple query that analyzes order data</span>
+<span class="c1">-- from Kafka and writes results into </span>
+<span class="c1">-- ElasticSearch. </span>
+<span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">product_analysis</span>
+<span class="k">SELECT</span>
+ <span class="n">product_id</span><span class="p">,</span>
+ <span class="n">TUMBLE_START</span><span class="p">(</span><span class="n">order_time</span><span class="p">,</span> <span class="nb">INTERVAL</span> <span class="s1">&#39;1&#39;</span> <span class="k">DAY</span><span class="p">)</span> <span class="k">as</span> <span class="n [...]
+ <span class="k">COUNT</span><span class="p">(</span><span class="o">*</span><span class="p">)</span> <span class="k">as</span> <span class="n">units_sold</span>
+<span class="k">FROM</span> <span class="n">orders</span>
+<span class="k">GROUP</span> <span class="k">BY</span>
+ <span class="n">product_id</span><span class="p">,</span>
+ <span class="n">TUMBLE</span><span class="p">(</span><span class="n">order_time</span><span class="p">,</span> <span class="nb">INTERVAL</span> <span class="s1">&#39;1&#39;</span> <span class="k">DAY</span><span class="p">);</span></code></pre></figure>
+
+<h1 id="catalogs">Catalogs</h1>
+
+<p>While being able to create tables is important, it often isn’t enough.
+A business analyst, for example, shouldn’t have to know what properties to set for Kafka, or even have to know what the underlying data source is, to be able to write a query.</p>
+
+<p>To solve this problem, Flink 1.10 also ships with a revamped catalog system for managing metadata about tables and user definined functions.
+With catalogs, users can create tables once and reuse them across Jobs and Sessions.
+Now, the team managing a data set can create a table and immediately make it accessible to other groups within their organization.</p>
+
+<p>The most notable catalog that Flink integrates with today is Hive Metastore.
+The Hive catalog allows Flink to fully interoperate with Hive and serve as a more efficient query engine.
+Flink supports reading and writing Hive tables, using Hive UDFs, and even leveraging Hive’s metastore catalog to persist Flink specific metadata.</p>
+
+<h1 id="looking-ahead">Looking Ahead</h1>
+
+<p>Flink SQL has made enormous strides to democratize stream processing, and 1.10 marks a significant milestone in that development.
+However, we are not ones to rest on our laurels and, the community is committed to raising the bar on standards while lowering the barriers to entry.
+The community is looking to add more catalogs, such as JDBC and Apache Pulsar.
+We encourage you to sign up for the <a href="https://flink.apache.org/community.html">mailing list</a> and stay on top of the announcements and new features in upcoming releases.</p>
+
+<hr />
+
+<div class="footnotes">
+ <ol>
+ <li id="fn:1">
+ <p>My colleague Timo, whose worked on Flink SQL from the beginning, has the entire SQL standard printed on his desk and references it before any changes are merged. It’s enormous. <a href="#fnref:1" class="reversefootnote">&#8617;</a></p>
+ </li>
+ </ol>
+</div>
+</description>
+<pubDate>Thu, 20 Feb 2020 13:00:00 +0100</pubDate>
+<link>https://flink.apache.org/news/2020/02/20/ddl.html</link>
+<guid isPermaLink="true">/news/2020/02/20/ddl.html</guid>
+</item>
+
+<item>
+<title>Apache Flink 1.10.0 Release Announcement</title>
+<description><p>The Apache Flink community is excited to hit the double digits and announce the release of Flink 1.10.0! As a result of the biggest community effort to date, with over 1.2k issues implemented and more than 200 contributors, this release introduces significant improvements to the overall performance and stability of Flink jobs, a preview of native Kubernetes integration and great advances in Python support (PyFlink).</p>
+
+<p>Flink 1.10 also marks the completion of the <a href="https://flink.apache.org/news/2019/08/22/release-1.9.0.html#preview-of-the-new-blink-sql-query-processor">Blink integration</a>, hardening streaming SQL and bringing mature batch processing to Flink with production-ready Hive integration and TPC-DS coverage. This blog post describes all major new features and improvements, important changes to be aware of and what to expect moving forward.</p>
+
+<div class="page-toc">
+<ul id="markdown-toc">
+ <li><a href="#new-features-and-improvements" id="markdown-toc-new-features-and-improvements">New Features and Improvements</a> <ul>
+ <li><a href="#improved-memory-management-and-configuration" id="markdown-toc-improved-memory-management-and-configuration">Improved Memory Management and Configuration</a></li>
+ <li><a href="#unified-logic-for-job-submission" id="markdown-toc-unified-logic-for-job-submission">Unified Logic for Job Submission</a></li>
+ <li><a href="#native-kubernetes-integration-beta" id="markdown-toc-native-kubernetes-integration-beta">Native Kubernetes Integration (Beta)</a></li>
+ <li><a href="#table-apisql-production-ready-hive-integration" id="markdown-toc-table-apisql-production-ready-hive-integration">Table API/SQL: Production-ready Hive Integration</a></li>
+ <li><a href="#other-improvements-to-the-table-apisql" id="markdown-toc-other-improvements-to-the-table-apisql">Other Improvements to the Table API/SQL</a></li>
+ <li><a href="#pyflink-support-for-native-user-defined-functions-udfs" id="markdown-toc-pyflink-support-for-native-user-defined-functions-udfs">PyFlink: Support for Native User Defined Functions (UDFs)</a></li>
+ </ul>
+ </li>
+ <li><a href="#important-changes" id="markdown-toc-important-changes">Important Changes</a></li>
+ <li><a href="#release-notes" id="markdown-toc-release-notes">Release Notes</a></li>
+ <li><a href="#list-of-contributors" id="markdown-toc-list-of-contributors">List of Contributors</a></li>
+</ul>
+
+</div>
+
+<p>The binary distribution and source artifacts are now available on the updated <a href="/downloads.html">Downloads page</a> of the Flink website. For more details, check the complete <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&amp;version=12345845">release changelog</a> and the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/">updated documentation</a> [...]
+
+<h2 id="new-features-and-improvements">New Features and Improvements</h2>
+
+<h3 id="improved-memory-management-and-configuration">Improved Memory Management and Configuration</h3>
+
+<p>The current <code>TaskExecutor</code> memory configuration in Flink has some shortcomings that make it hard to reason about or optimize resource utilization, such as:</p>
+
+<ul>
+ <li>
+ <p>Different configuration models for memory footprint in Streaming and Batch execution;</p>
+ </li>
+ <li>
+ <p>Complex and user-dependent configuration of off-heap state backends (i.e. RocksDB) in Streaming execution.</p>
+ </li>
+</ul>
+
+<p>To make memory options more explicit and intuitive to users, Flink 1.10 introduces significant changes to the <code>TaskExecutor</code> memory model and configuration logic (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors">FLIP-49</a>). These changes make Flink more adaptable to all kinds of deployment environments (e.g. Kubernetes, Yarn, Mesos), giving users strict control ove [...]
+
+<p><strong>Managed Memory Extension</strong></p>
+
+<p>Managed memory was extended to also account for memory usage of <code>RocksDBStateBackend</code>. While batch jobs can use either on-heap or off-heap memory, streaming jobs with <code>RocksDBStateBackend</code> can use off-heap memory only. Therefore, to allow users to switch between Streaming and Batch execution without having to modify cluster configurations, managed memory is now always off-heap.</p>
+
+<p><strong>Simplified RocksDB Configuration</strong></p>
+
+<p>Configuring an off-heap state backend like RocksDB used to involve a good deal of manual tuning, like decreasing the JVM heap size or setting Flink to use off-heap memory. This can now be achieved through Flink’s out-of-box configuration, and adjusting the memory budget for <code>RocksDBStateBackend</code> is as simple as resizing the managed memory size.</p>
+
+<p>Another important improvement was to allow Flink to bind RocksDB native memory usage (<a href="https://issues.apache.org/jira/browse/FLINK-7289">FLINK-7289</a>), preventing it from exceeding its total memory budget — this is especially relevant in containerized environments like Kubernetes. For details on how to enable and tune this feature, refer to <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/state/large_state_tuning.h [...]
+
+<p><span class="label label-danger">Note</span> FLIP-49 changes the process of cluster resource configuration, which may require tuning your clusters for upgrades from previous Flink versions. For a comprehensive overview of the changes introduced and tuning guidance, consult <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/memory/mem_setup.html">this setup</a>.</p>
+
+<h3 id="unified-logic-for-job-submission">Unified Logic for Job Submission</h3>
+
+<p>Prior to this release, job submission was part of the duties of the Execution Environments and closely tied to the different deployment targets (e.g. Yarn, Kubernetes, Mesos). This led to a poor separation of concerns and, over time, to a growing number of customized environments that users needed to configure and manage separately.</p>
+
+<p>In Flink 1.10, job submission logic is abstracted into the generic <code>Executor</code> interface (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-73%3A+Introducing+Executors+for+job+submission">FLIP-73</a>). The addition of the <code>ExecutorCLI</code> (<a href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=133631524">FLIP-81</a>) introduces a unified way to specify configura [...]
+
+<p><span>
+ <center>
+ <img vspace="8" style="width:100%" src="/img/blog/2020-02-11-release-1.10.0/flink_1.10_zeppelin.png" />
+ </center>
+</span></p>
+
+<p>In particular, these changes make it much easier to programmatically use Flink in downstream frameworks — for example, Apache Beam or Zeppelin interactive notebooks — by providing users with a unified entry point to Flink. For users working with Flink across multiple target environments, the transition to a configuration-based execution process also significantly reduces boilerplate code and maintainability overhead.</p>
+
+<h3 id="native-kubernetes-integration-beta">Native Kubernetes Integration (Beta)</h3>
+
+<p>For users looking to get started with Flink on a containerized environment, deploying and managing a standalone cluster on top of Kubernetes requires some upfront knowledge about containers, operators and environment-specific tools like <code>kubectl</code>.</p>
+
+<p>In Flink 1.10, we rolled out the first phase of <strong>Active Kubernetes Integration</strong> (<a href="https://jira.apache.org/jira/browse/FLINK-9953">FLINK-9953</a>) with support for session clusters (with per-job planned). In this context, “active” means that Flink’s ResourceManager (<code>K8sResMngr</code>) natively communicates with Kubernetes to allocate new pods on-demand, similar to Flink’s Yarn and Mesos integration. Users [...]
+
+<p><span>
+ <center>
+ <img vspace="8" style="width:75%" src="/img/blog/2020-02-11-release-1.10.0/flink_1.10_nativek8s.png" />
+ </center>
+</span></p>
+
+<p>As introduced in <a href="#unified-logic-for-job-submission">Unified Logic For Job Submission</a>, all command-line options in Flink 1.10 are mapped to a unified configuration. For this reason, users can simply refer to the Kubernetes config options and submit a job to an existing Flink session on Kubernetes in the CLI using:</p>
+
+<div class="highlight"><pre><code class="language-bash">./bin/flink run -d -e kubernetes-session -Dkubernetes.cluster-id<span class="o">=</span>&lt;ClusterId&gt; examples/streaming/WindowJoin.jar</code></pre></div>
+
+<p>If you want to try out this preview feature, we encourage you to walk through the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/deployment/native_kubernetes.html">Native Kubernetes setup</a>, play around with it and share feedback with the community.</p>
+
+<h3 id="table-apisql-production-ready-hive-integration">Table API/SQL: Production-ready Hive Integration</h3>
+
+<p>Hive integration was announced as a preview feature in Flink 1.9. This preview allowed users to persist Flink-specific metadata (e.g. Kafka tables) in Hive Metastore using SQL DDL, call UDFs defined in Hive and use Flink for reading and writing Hive tables. Flink 1.10 rounds up this effort with further developments that bring production-ready Hive integration to Flink with full compatibility of <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/tab [...]
+
+<h4 id="native-partition-support-for-batch-sql">Native Partition Support for Batch SQL</h4>
+
+<p>So far, only writes to non-partitioned Hive tables were supported. In Flink 1.10, the Flink SQL syntax has been extended with <code>INSERT OVERWRITE</code> and <code>PARTITION</code> (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-63%3A+Rework+table+partition+support">FLIP-63</a>), enabling users to write into both static and dynamic partitions in Hive.</p>
+
+<p><strong>Static Partition Writing</strong></p>
+
+<div class="highlight"><pre><code class="language-sql"><span class="k">INSERT</span> <span class="err">{</span> <span class="k">INTO</span> <span class="o">|</span> <span class="n">OVERWRITE</span> <span class="err">}</span> <span class="k">TABLE</span> <span class="n">tablename1< [...]
+
+<p><strong>Dynamic Partition Writing</strong></p>
+
+<div class="highlight"><pre><code class="language-sql"><span class="k">INSERT</span> <span class="err">{</span> <span class="k">INTO</span> <span class="o">|</span> <span class="n">OVERWRITE</span> <span class="err">}</span> <span class="k">TABLE</span> <span class="n">tablename1< [...]
+
+<p>Fully supporting partitioned tables allows users to take advantage of partition pruning on read, which significantly increases the performance of these operations by reducing the amount of data that needs to be scanned.</p>
+
+<h4 id="further-optimizations">Further Optimizations</h4>
+
+<p>Besides partition pruning, Flink 1.10 introduces more <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/hive/read_write_hive.html#optimizations">read optimizations</a> to Hive integration, such as:</p>
+
+<ul>
+ <li>
+ <p><strong>Projection pushdown:</strong> Flink leverages projection pushdown to minimize data transfer between Flink and Hive tables by omitting unnecessary fields from table scans. This is especially beneficial for tables with a large number of columns.</p>
+ </li>
+ <li>
+ <p><strong>LIMIT pushdown:</strong> for queries with the <code>LIMIT</code> clause, Flink will limit the number of output records wherever possible to minimize the amount of data transferred across the network.</p>
+ </li>
+ <li>
+ <p><strong>ORC Vectorization on Read:</strong> to boost read performance for ORC files, Flink now uses the native ORC Vectorized Reader by default for Hive versions above 2.0.0 and columns with non-complex data types.</p>
+ </li>
+</ul>
+
+<h4 id="pluggable-modules-as-flink-system-objects-beta">Pluggable Modules as Flink System Objects (Beta)</h4>
+
+<p>Flink 1.10 introduces a generic mechanism for pluggable modules in the Flink table core, with a first focus on system functions (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-68%3A+Extend+Core+Table+System+with+Pluggable+Modules">FLIP-68</a>). With modules, users can extend Flink’s system objects — for example use Hive built-in functions that behave like Flink system functions. This release ships with a pre-implemented <code>HiveModu [...]
+
+<h3 id="other-improvements-to-the-table-apisql">Other Improvements to the Table API/SQL</h3>
+
+<h4 id="watermarks-and-computed-columns-in-sql-ddl">Watermarks and Computed Columns in SQL DDL</h4>
+
+<p>Flink 1.10 supports stream-specific syntax extensions to define time attributes and watermark generation in Flink SQL DDL (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-66%3A+Support+Time+Attribute+in+SQL+DDL">FLIP-66</a>). This allows time-based operations, like windowing, and the definition of <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/sql/create.html#create-table">watermark strategies [...]
+
+<div class="highlight"><pre><code class="language-sql"><span class="k">CREATE</span> <span class="k">TABLE</span> <span class="k">table_name</span> <span class="p">(</span>
+
+ <span class="n">WATERMARK</span> <span class="k">FOR</span> <span class="n">columnName</span> <span class="k">AS</span> <span class="o">&lt;</span><span class="n">watermark_strategy_expression</span><span class="o">&gt;</span>
+
+<span class="p">)</span> <span class="k">WITH</span> <span class="p">(</span>
+ <span class="p">...</span>
+<span class="p">)</span></code></pre></div>
+
+<p>This release also introduces support for virtual computed columns (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-70%3A+Flink+SQL+Computed+Column+Design">FLIP-70</a>) that can be derived based on other columns in the same table or deterministic expressions (i.e. literal values, UDFs and built-in functions). In Flink, computed columns are useful to define time attributes <a href="https://ci.apache.org/projects/flink/flink-docs-releas [...]
+
+<h4 id="additional-extensions-to-sql-ddl">Additional Extensions to SQL DDL</h4>
+
+<p>There is now a clear distinction between temporary/persistent and system/catalog functions (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-57%3A+Rework+FunctionCatalog">FLIP-57</a>). This not only eliminates ambiguity in function reference, but also allows for deterministic function resolution order (i.e. in case of naming collision, system functions will precede catalog functions, with temporary functions taking precedence over persistent [...]
+
+<p>Following the groundwork in FLIP-57, we extended the SQL DDL syntax to support the creation of catalog functions, temporary functions and temporary system functions (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-79+Flink+Function+DDL+Support">FLIP-79</a>):</p>
+
+<div class="highlight"><pre><code class="language-sql"><span class="k">CREATE</span> <span class="p">[</span><span class="k">TEMPORARY</span><span class="o">|</span><span class="k">TEMPORARY</span> <span class="k">SYSTEM</span><span class="p">]</span> <span class="k">FUNCTION</span>
+
+ <span class="p">[</span><span class="n">IF</span> <span class="k">NOT</span> <span class="k">EXISTS</span><span class="p">]</span> <span class="p">[</span><span class="k">catalog_name</span><span class="p">.][</span><span class="n">db_name</span><span class="p">.]</span> [...]
+
+<span class="k">AS</span> <span class="n">identifier</span> <span class="p">[</span><span class="k">LANGUAGE</span> <span class="n">JAVA</span><span class="o">|</span><span class="n">SCALA</span><span class="p">]</span></code></pre></div>
+
+<p>For a complete overview of the current state of DDL support in Flink SQL, check the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/sql/">updated documentation</a>.</p>
+
+<p><span class="label label-danger">Note</span> In order to correctly handle and guarantee a consistent behavior across meta-objects (tables, views, functions) in the future, some object declaration methods in the Table API have been deprecated in favor of methods that are closer to standard SQL DDL (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-64%3A+Support+for+Temporary+Objects+in+Table+module">FLIP-64</a>).</p>
+
+<h4 id="full-tpc-ds-coverage-for-batch">Full TPC-DS Coverage for Batch</h4>
+
+<p>TPC-DS is a widely used industry-standard decision support benchmark to evaluate and measure the performance of SQL-based data processing engines. In Flink 1.10, all TPC-DS queries are supported end-to-end (<a href="https://issues.apache.org/jira/browse/FLINK-11491">FLINK-11491</a>), reflecting the readiness of its SQL engine to address the needs of modern data warehouse-like workloads.</p>
+
+<h3 id="pyflink-support-for-native-user-defined-functions-udfs">PyFlink: Support for Native User Defined Functions (UDFs)</h3>
+
+<p>A preview of PyFlink was introduced in the previous release, making headway towards the goal of full Python support in Flink. For this release, the focus was to enable users to register and use Python User-Defined Functions (UDF, with UDTF/UDAF planned) in the Table API/SQL (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-58%3A+Flink+Python+User-Defined+Stateless+Function+for+Table">FLIP-58</a>).</p>
+
+<p><span>
+ <center>
+ <img vspace="8" hspace="100" style="width:75%" src="/img/blog/2020-02-11-release-1.10.0/flink_1.10_pyflink.gif" />
+ </center>
+</span></p>
+
+<p>If you are interested in the underlying implementation — leveraging Apache Beam’s <a href="https://beam.apache.org/roadmap/portability/">Portability Framework</a> — refer to the “Architecture” section of FLIP-58 and also to <a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-78%3A+Flink+Python+UDF+Environment+and+Dependency+Management">FLIP-78</a>. These data structures lay the required foundation for Pandas support and for [...]
+
+<p>From Flink 1.10, users can also easily install PyFlink through <code>pip</code> using:</p>
+
+<div class="highlight"><pre><code class="language-bash">pip install apache-flink</code></pre></div>
+
+<p>For a preview of other improvements planned for PyFlink, check <a href="https://issues.apache.org/jira/browse/FLINK-14500">FLINK-14500</a> and get involved in the <a href="http://apache-flink.147419.n8.nabble.com/Re-DISCUSS-What-parts-of-the-Python-API-should-we-focus-on-next-td1285.html">discussion</a> for requested user features.</p>
+
+<h2 id="important-changes">Important Changes</h2>
+
+<ul>
+ <li>
+ <p>[<a href="https://issues.apache.org/jira/browse/FLINK-10725">FLINK-10725</a>] Flink can now be compiled and run on Java 11.</p>
+ </li>
+ <li>
+ <p>[<a href="https://jira.apache.org/jira/browse/FLINK-15495">FLINK-15495</a>] The Blink planner is now the default in the SQL Client, so that users can benefit from all the latest features and improvements. The switch from the old planner in the Table API is also planned for the next release, so we recommend that users start getting familiar with the Blink planner.</p>
+ </li>
+ <li>
+ <p>[<a href="https://issues.apache.org/jira/browse/FLINK-13025">FLINK-13025</a>] There is a <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/connectors/elasticsearch.html#elasticsearch-connector">new Elasticsearch sink connector</a>, fully supporting Elasticsearch 7.x versions.</p>
+ </li>
+ <li>
+ <p>[<a href="https://issues.apache.org/jira/browse/FLINK-15115">FLINK-15115</a>] The connectors for Kafka 0.8 and 0.9 have been marked as deprecated and will no longer be actively supported. If you are still using these versions or have any other related concerns, please reach out to the @dev mailing list.</p>
+ </li>
+ <li>
+ <p>[<a href="https://issues.apache.org/jira/browse/FLINK-14516">FLINK-14516</a>] The non-credit-based network flow control code was removed, along with the configuration option <code>taskmanager.network.credit.model</code>. Moving forward, Flink will always use credit-based flow control.</p>
+ </li>
+ <li>
+ <p>[<a href="https://issues.apache.org/jira/browse/FLINK-12122">FLINK-12122</a>] <a href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65147077">FLIP-6</a> was rolled out with Flink 1.5.0 and introduced a code regression related to the way slots are allocated from <code>TaskManagers</code>. To use a scheduling strategy that is closer to the pre-FLIP behavior, where Flink tries to spread out the workload [...]
+ </li>
+ <li>
+ <p>[<a href="https://issues.apache.org/jira/browse/FLINK-11956">FLINK-11956</a>] <code>s3-hadoop</code> and <code>s3-presto</code> filesystems no longer use class relocations and should be loaded through <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/filesystems/#pluggable-file-systems">plugins</a>, but now seamlessly integrate with all credential providers. Other filesystems are stro [...]
+ </li>
+ <li>
+ <p>Flink 1.9 shipped with a refactored Web UI, with the legacy one being kept around as backup in case something wasn’t working as expected. No issues have been reported so far, so <a href="http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Remove-old-WebUI-td35218.html">the community voted</a> to drop the legacy Web UI in Flink 1.10.</p>
+ </li>
+</ul>
+
+<h2 id="release-notes">Release Notes</h2>
+
+<p>Please review the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.10/release-notes/flink-1.10.html">release notes</a> carefully for a detailed list of changes and new features if you plan to upgrade your setup to Flink 1.10. This version is API-compatible with previous 1.x releases for APIs annotated with the @Public annotation.</p>
+
+<h2 id="list-of-contributors">List of Contributors</h2>
+
+<p>The Apache Flink community would like to thank all contributors that have made this release possible:</p>
+
+<p>Achyuth Samudrala, Aitozi, Alberto Romero, Alec.Ch, Aleksey Pak, Alexander Fedulov, Alice Yan, Aljoscha Krettek, Aloys, Andrey Zagrebin, Arvid Heise, Benchao Li, Benoit Hanotte, Benoît Paris, Bhagavan Das, Biao Liu, Chesnay Schepler, Congxian Qiu, Cyrille Chépélov, César Soto Valero, David Anderson, David Hrbacek, David Moravek, Dawid Wysakowicz, Dezhi Cai, Dian Fu, Dyana Rose, Eamon Taaffe, Fabian Hueske, Fawad Halim, Fokko Driesprong, Frey Gao, Gabor Gevay, Gao Yun, Gary Yao, [...]
+</description>
+<pubDate>Tue, 11 Feb 2020 03:30:00 +0100</pubDate>
+<link>https://flink.apache.org/news/2020/02/11/release-1.10.0.html</link>
+<guid isPermaLink="true">/news/2020/02/11/release-1.10.0.html</guid>
+</item>
+
+<item>
+<title>A Guide for Unit Testing in Apache Flink</title>
+<description><p>Writing unit tests is one of the essential tasks of designing a production-grade application. Without tests, a single change in code can result in cascades of failure in production. Thus unit tests should be written for all types of applications, be it a simple job cleaning data and training a model or a complex multi-tenant, real-time data processing system. In the following sections, we provide a guide for unit testing of Apache Flink applications.
+Apache Flink provides a robust unit testing framework to make sure your applications behave in production as expected during development. You need to include the following dependencies to utilize the provided framework.</p>
+
+<div class="highlight"><pre><code class="language-xml"><span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-test-utils_${scala.binary.version}<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>${flink.version}<span class="nt">&lt;/version&gt;</span>
+ <span class="nt">&lt;scope&gt;</span>test<span class="nt">&lt;/scope&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span>
+<span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-runtime_2.11<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.9.0<span class="nt">&lt;/version&gt;</span>
+ <span class="nt">&lt;scope&gt;</span>test<span class="nt">&lt;/scope&gt;</span>
+ <span class="nt">&lt;classifier&gt;</span>tests<span class="nt">&lt;/classifier&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span>
+<span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-streaming-java_2.11<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.9.0<span class="nt">&lt;/version&gt;</span>
+ <span class="nt">&lt;scope&gt;</span>test<span class="nt">&lt;/scope&gt;</span>
+ <span class="nt">&lt;classifier&gt;</span>tests<span class="nt">&lt;/classifier&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span></code></pre></div>
+
+<p>The strategy of writing unit tests differs for various operators. You can break down the strategy into the following three buckets:</p>
+
+<ul>
+ <li>Stateless Operators</li>
+ <li>Stateful Operators</li>
+ <li>Timed Process Operators</li>
+</ul>
+
+<h1 id="stateless-operators">Stateless Operators</h1>
+
+<p>Writing unit tests for a stateless operator is a breeze. You need to follow the basic norm of writing a test case, i.e., create an instance of the function class and test the appropriate methods. Let’s take an example of a simple <code>Map</code> operator.</p>
+
+<div class="highlight"><pre><code class="language-java"><span class="kd">public</span> <span class="kd">class</span> <span class="nc">MyStatelessMap</span> <span class="kd">implements</span> <span class="n">MapFunction</span><span class="o">&lt;</span><span class="n">String</span><span class= [...]
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="n">String</span> <span class="nf">map</span><span class="o">(</span><span class="n">String</span> <span class="n">in</span><span class="o">)</span> <span class="kd">throws</span> <span class="n">Exception</span> <span class="o">{< [...]
+ <span class="n">String</span> <span class="n">out</span> <span class="o">=</span> <span class="s">&quot;hello &quot;</span> <span class="o">+</span> <span class="n">in</span><span class="o">;</span>
+ <span class="k">return</span> <span class="n">out</span><span class="o">;</span>
+ <span class="o">}</span>
+<span class="o">}</span></code></pre></div>
+
+<p>The test case for the above operator should look like</p>
+
+<div class="highlight"><pre><code class="language-java"><span class="nd">@Test</span>
+<span class="kd">public</span> <span class="kt">void</span> <span class="nf">testMap</span><span class="o">()</span> <span class="kd">throws</span> <span class="n">Exception</span> <span class="o">{</span>
+ <span class="n">MyStatelessMap</span> <span class="n">statelessMap</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">MyStatelessMap</span><span class="o">();</span>
+ <span class="n">String</span> <span class="n">out</span> <span class="o">=</span> <span class="n">statelessMap</span><span class="o">.</span><span class="na">map</span><span class="o">(</span><span class="s">&quot;world&quot;</span><span class="o">);</span>
+ <span class="n">Assert</span><span class="o">.</span><span class="na">assertEquals</span><span class="o">(</span><span class="s">&quot;hello world&quot;</span><span class="o">,</span> <span class="n">out</span><span class="o">);</span>
+<span class="o">}</span></code></pre></div>
+
+<p>Pretty simple, right? Let’s take a look at one for the <code>FlatMap</code> operator.</p>
+
+<div class="highlight"><pre><code class="language-java"><span class="kd">public</span> <span class="kd">class</span> <span class="nc">MyStatelessFlatMap</span> <span class="kd">implements</span> <span class="n">FlatMapFunction</span><span class="o">&lt;</span><span class="n">String</span><spa [...]
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="nf">flatMap</span><span class="o">(</span><span class="n">String</span> <span class="n">in</span><span class="o">,</span> <span class="n">Collector</span><span class="o">&lt;</span><span class="n">St [...]
+ <span class="n">String</span> <span class="n">out</span> <span class="o">=</span> <span class="s">&quot;hello &quot;</span> <span class="o">+</span> <span class="n">in</span><span class="o">;</span>
+ <span class="n">collector</span><span class="o">.</span><span class="na">collect</span><span class="o">(</span><span class="n">out</span><span class="o">);</span>
+ <span class="o">}</span>
+<span class="o">}</span></code></pre></div>
+
+<p><code>FlatMap</code> operators require a <code>Collector</code> object along with the input. For the test case, we have two options:</p>
+
+<ol>
+ <li>Mock the <code>Collector</code> object using Mockito</li>
+ <li>Use the <code>ListCollector</code> provided by Flink</li>
+</ol>
+
+<p>I prefer the second method as it requires fewer lines of code and is suitable for most of the cases.</p>
+
+<div class="highlight"><pre><code class="language-java"><span class="nd">@Test</span>
+<span class="kd">public</span> <span class="kt">void</span> <span class="nf">testFlatMap</span><span class="o">()</span> <span class="kd">throws</span> <span class="n">Exception</span> <span class="o">{</span>
+ <span class="n">MyStatelessFlatMap</span> <span class="n">statelessFlatMap</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">MyStatelessFlatMap</span><span class="o">();</span>
+ <span class="n">List</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">out</span> <span class="o">=</span> <span class="k">new</span> <span class="n">ArrayList</span><span class="o">&lt;&gt;();</span>
+ <span class="n">ListCollector</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">listCollector</span> <span class="o">=</span> <span class="k">new</span> <span class="n">ListCollector</span><span class="o">&lt;&gt;(</span> [...]
+ <span class="n">statelessFlatMap</span><span class="o">.</span><span class="na">flatMap</span><span class="o">(</span><span class="s">&quot;world&quot;</span><span class="o">,</span> <span class="n">listCollector</span><span class="o">);</span>
+ <span class="n">Assert</span><span class="o">.</span><span class="na">assertEquals</span><span class="o">(</span><span class="n">Lists</span><span class="o">.</span><span class="na">newArrayList</span><span class="o">(</span><span class="s">&quot;hello world&quot;</span><span clas [...]
+<span class="o">}</span></code></pre></div>
+
+<h1 id="stateful-operators">Stateful Operators</h1>
+
+<p>Writing test cases for stateful operators requires more effort. You need to check whether the operator state is updated correctly and if it is cleaned up properly along with the output of the operator.</p>
+
+<p>Let’s take an example of stateful <code>FlatMap</code> function</p>
+
+<div class="highlight"><pre><code class="language-java"><span class="kd">public</span> <span class="kd">class</span> <span class="nc">StatefulFlatMap</span> <span class="kd">extends</span> <span class="n">RichFlatMapFunction</span><span class="o">&lt;</span><span class="n">String</span><span [...]
+ <span class="n">ValueState</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">previousInput</span><span class="o">;</span>
+
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="nf">open</span><span class="o">(</span><span class="n">Configuration</span> <span class="n">parameters</span><span class="o">)</span> <span class="kd">throws</span> <span class="n">Exception</span> <span class=" [...]
+ <span class="n">previousInput</span> <span class="o">=</span> <span class="n">getRuntimeContext</span><span class="o">().</span><span class="na">getState</span><span class="o">(</span>
+ <span class="k">new</span> <span class="n">ValueStateDescriptor</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;(</span><span class="s">&quot;previousInput&quot;</span><span class="o">,</span> <span class="n">Types</span><span class="o">.</s [...]
+ <span class="o">}</span>
+
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="nf">flatMap</span><span class="o">(</span><span class="n">String</span> <span class="n">in</span><span class="o">,</span> <span class="n">Collector</span><span class="o">&lt;</span><span class="n">St [...]
+ <span class="n">String</span> <span class="n">out</span> <span class="o">=</span> <span class="s">&quot;hello &quot;</span> <span class="o">+</span> <span class="n">in</span><span class="o">;</span>
+ <span class="k">if</span><span class="o">(</span><span class="n">previousInput</span><span class="o">.</span><span class="na">value</span><span class="o">()</span> <span class="o">!=</span> <span class="kc">null</span><span class="o">){</span>
+ <span class="n">out</span> <span class="o">=</span> <span class="n">out</span> <span class="o">+</span> <span class="s">&quot; &quot;</span> <span class="o">+</span> <span class="n">previousInput</span><span class="o">.</span><span class="na">value</span><span class="o&qu [...]
+ <span class="o">}</span>
+ <span class="n">previousInput</span><span class="o">.</span><span class="na">update</span><span class="o">(</span><span class="n">in</span><span class="o">);</span>
+ <span class="n">collector</span><span class="o">.</span><span class="na">collect</span><span class="o">(</span><span class="n">out</span><span class="o">);</span>
+ <span class="o">}</span>
+<span class="o">}</span></code></pre></div>
+
+<p>The intricate part of writing tests for the above class is to mock the configuration as well as the runtime context of the application. Flink provides TestHarness classes so that users don’t have to create the mock objects themselves. Using the <code>KeyedOperatorHarness</code>, the test looks like:</p>
+
+<div class="highlight"><pre><code class="language-java"><span class="kn">import</span> <span class="nn">org.apache.flink.streaming.api.operators.StreamFlatMap</span><span class="o">;</span>
+<span class="kn">import</span> <span class="nn">org.apache.flink.streaming.runtime.streamrecord.StreamRecord</span><span class="o">;</span>
+<span class="kn">import</span> <span class="nn">org.apache.flink.streaming.util.KeyedOneInputStreamOperatorTestHarness</span><span class="o">;</span>
+<span class="kn">import</span> <span class="nn">org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness</span><span class="o">;</span>
+
+<span class="nd">@Test</span>
+<span class="kd">public</span> <span class="kt">void</span> <span class="nf">testFlatMap</span><span class="o">()</span> <span class="kd">throws</span> <span class="n">Exception</span><span class="o">{</span>
+ <span class="n">StatefulFlatMap</span> <span class="n">statefulFlatMap</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">StatefulFlatMap</span><span class="o">();</span>
+
+ <span class="c1">// OneInputStreamOperatorTestHarness takes the input and output types as type parameters </span>
+ <span class="n">OneInputStreamOperatorTestHarness</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">String</span><span class="o">&gt;</span> <span class="n">testHarness</span> <span class="o">=</span>
+ <span class="c1">// KeyedOneInputStreamOperatorTestHarness takes three arguments:</span>
+ <span class="c1">// Flink operator object, key selector and key type</span>
+ <span class="k">new</span> <span class="n">KeyedOneInputStreamOperatorTestHarness</span><span class="o">&lt;&gt;(</span>
+ <span class="k">new</span> <span class="n">StreamFlatMap</span><span class="o">&lt;&gt;(</span><span class="n">statefulFlatMap</span><span class="o">),</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="s">&quot;1&quot;</span><span class="o">,</spa [...]
+ <span class="n">testHarness</span><span class="o">.</span><span class="na">open</span><span class="o">();</span>
+
+ <span class="c1">// test first record</span>
+ <span class="n">testHarness</span><span class="o">.</span><span class="na">processElement</span><span class="o">(</span><span class="s">&quot;world&quot;</span><span class="o">,</span> <span class="mi">10</span><span class="o">);</span>
+ <span class="n">ValueState</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">previousInput</span> <span class="o">=</span>
+ <span class="n">statefulFlatMap</span><span class="o">.</span><span class="na">getRuntimeContext</span><span class="o">().</span><span class="na">getState</span><span class="o">(</span>
+ <span class="k">new</span> <span class="n">ValueStateDescriptor</span><span class="o">&lt;&gt;(</span><span class="s">&quot;previousInput&quot;</span><span class="o">,</span> <span class="n">Types</span><span class="o">.</span><span class="na">STRING</span><span class="o">));&l [...]
+ <span class="n">String</span> <span class="n">stateValue</span> <span class="o">=</span> <span class="n">previousInput</span><span class="o">.</span><span class="na">value</span><span class="o">();</span>
+ <span class="n">Assert</span><span class="o">.</span><span class="na">assertEquals</span><span class="o">(</span>
+ <span class="n">Lists</span><span class="o">.</span><span class="na">newArrayList</span><span class="o">(</span><span class="k">new</span> <span class="n">StreamRecord</span><span class="o">&lt;&gt;(</span><span class="s">&quot;hello world&quot;</span><span class="o">,</span&g [...]
+ <span class="n">testHarness</span><span class="o">.</span><span class="na">extractOutputStreamRecords</span><span class="o">());</span>
+ <span class="n">Assert</span><span class="o">.</span><span class="na">assertEquals</span><span class="o">(</span><span class="s">&quot;world&quot;</span><span class="o">,</span> <span class="n">stateValue</span><span class="o">);</span>
+
+ <span class="c1">// test second record</span>
+ <span class="n">testHarness</span><span class="o">.</span><span class="na">processElement</span><span class="o">(</span><span class="s">&quot;parallel&quot;</span><span class="o">,</span> <span class="mi">20</span><span class="o">);</span>
+ <span class="n">Assert</span><span class="o">.</span><span class="na">assertEquals</span><span class="o">(</span>
+ <span class="n">Lists</span><span class="o">.</span><span class="na">newArrayList</span><span class="o">(</span>
+ <span class="k">new</span> <span class="n">StreamRecord</span><span class="o">&lt;&gt;(</span><span class="s">&quot;hello world&quot;</span><span class="o">,</span> <span class="mi">10</span><span class="o">),</span>
+ <span class="k">new</span> <span class="n">StreamRecord</span><span class="o">&lt;&gt;(</span><span class="s">&quot;hello parallel world&quot;</span><span class="o">,</span> <span class="mi">20</span><span class="o">)),</span>
+ <span class="n">testHarness</span><span class="o">.</span><span class="na">extractOutputStreamRecords</span><span class="o">());</span>
+ <span class="n">Assert</span><span class="o">.</span><span class="na">assertEquals</span><span class="o">(</span><span class="s">&quot;parallel&quot;</span><span class="o">,</span> <span class="n">previousInput</span><span class="o">.</span><span class="na">value</span><span class [...]
+<span class="o">}</span></code></pre></div>
+
+<p>The test harness provides many helper methods, three of which are being used here:</p>
+
+<ol>
+ <li><code>open</code>: calls the open of the <code>FlatMap</code> function with relevant parameters. It also initializes the context.</li>
+ <li><code>processElement</code>: allows users to pass an input element as well as the timestamp associated with the element.</li>
+ <li><code>extractOutputStreamRecords</code>: gets the output records along with their timestamps from the <code>Collector</code>.</li>
+</ol>
+
+<p>The test harness simplifies the unit testing for the stateful functions to a large extent.</p>
+
+<p>You might also need to check whether the state value is being set correctly. You can get the state value directly from the operator using a mechanism similar to the one used while creating the state. This is also demonstrated in the previous example.</p>
+
+<h1 id="timed-process-operators">Timed Process Operators</h1>
+
+<p>Writing tests for process functions, that work with time, is quite similar to writing tests for stateful functions because you can also use test harness.
+However, you need to take care of another aspect, which is providing timestamps for events and controlling the current time of the application. By setting the current (processing or event) time, you can trigger registered timers, which will call the <code>onTimer</code> method of the function</p>
+
+<div class="highlight"><pre><code class="language-java"><span class="kd">public</span> <span class="kd">class</span> <span class="nc">MyProcessFunction</span> <span class="kd">extends</span> <span class="n">KeyedProcessFunction</span><span class="o">&lt;</span><span class="n">String</span><sp [...]
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="nf">processElement</span><span class="o">(</span><span class="n">String</span> <span class="n">in</span><span class="o">,</span> <span class="n">Context</span> <span class="n">context</span><span class="o"& [...]
+ <span class="n">context</span><span class="o">.</span><span class="na">timerService</span><span class="o">().</span><span class="na">registerProcessingTimeTimer</span><span class="o">(</span><span class="mi">50</span><span class="o">);</span>
+ <span class="n">String</span> <span class="n">out</span> <span class="o">=</span> <span class="s">&quot;hello &quot;</span> <span class="o">+</span> <span class="n">in</span><span class="o">;</span>
+ <span class="n">collector</span><span class="o">.</span><span class="na">collect</span><span class="o">(</span><span class="n">out</span><span class="o">);</span>
+ <span class="o">}</span>
+
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="nf">onTimer</span><span class="o">(</span><span class="kt">long</span> <span class="n">timestamp</span><span class="o">,</span> <span class="n">OnTimerContext</span> <span class="n">ctx</span><span class="o" [...]
+ <span class="n">out</span><span class="o">.</span><span class="na">collect</span><span class="o">(</span><span class="n">String</span><span class="o">.</span><span class="na">format</span><span class="o">(</span><span class="s">&quot;Timer triggered at timestamp %d&quot;</span>< [...]
+ <span class="o">}</span>
+<span class="o">}</span></code></pre></div>
+
+<p>We need to test both the methods in the <code>KeyedProcessFunction</code>, i.e., <code>processElement</code> as well as <code>onTimer</code>. Using a test harness, we can control the current time of the function. Thus, we can trigger the timer at will rather than waiting for a specific time.</p>
+
+<p>Let’s take a look at the test case</p>
+
+<div class="highlight"><pre><code class="language-java"><span class="nd">@Test</span>
+<span class="kd">public</span> <span class="kt">void</span> <span class="nf">testProcessElement</span><span class="o">()</span> <span class="kd">throws</span> <span class="n">Exception</span><span class="o">{</span>
+ <span class="n">MyProcessFunction</span> <span class="n">myProcessFunction</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">MyProcessFunction</span><span class="o">();</span>
+ <span class="n">OneInputStreamOperatorTestHarness</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">String</span><span class="o">&gt;</span> <span class="n">testHarness</span> <span class="o">=</span>
+ <span class="k">new</span> <span class="n">KeyedOneInputStreamOperatorTestHarness</span><span class="o">&lt;&gt;(</span>
+ <span class="k">new</span> <span class="n">KeyedProcessOperator</span><span class="o">&lt;&gt;(</span><span class="n">myProcessFunction</span><span class="o">),</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="s">&quot;1&quot;</span><span class="o"> [...]
+
+ <span class="c1">// Function time is initialized to 0</span>
+ <span class="n">testHarness</span><span class="o">.</span><span class="na">open</span><span class="o">();</span>
+ <span class="n">testHarness</span><span class="o">.</span><span class="na">processElement</span><span class="o">(</span><span class="s">&quot;world&quot;</span><span class="o">,</span> <span class="mi">10</span><span class="o">);</span>
+
+ <span class="n">Assert</span><span class="o">.</span><span class="na">assertEquals</span><span class="o">(</span>
+ <span class="n">Lists</span><span class="o">.</span><span class="na">newArrayList</span><span class="o">(</span><span class="k">new</span> <span class="n">StreamRecord</span><span class="o">&lt;&gt;(</span><span class="s">&quot;hello world&quot;</span><span class="o">,</span&g [...]
+ <span class="n">testHarness</span><span class="o">.</span><span class="na">extractOutputStreamRecords</span><span class="o">());</span>
+<span class="o">}</span>
+
+<span class="nd">@Test</span>
+<span class="kd">public</span> <span class="kt">void</span> <span class="nf">testOnTimer</span><span class="o">()</span> <span class="kd">throws</span> <span class="n">Exception</span> <span class="o">{</span>
+ <span class="n">MyProcessFunction</span> <span class="n">myProcessFunction</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">MyProcessFunction</span><span class="o">();</span>
+ <span class="n">OneInputStreamOperatorTestHarness</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">String</span><span class="o">&gt;</span> <span class="n">testHarness</span> <span class="o">=</span>
+ <span class="k">new</span> <span class="n">KeyedOneInputStreamOperatorTestHarness</span><span class="o">&lt;&gt;(</span>
+ <span class="k">new</span> <span class="n">KeyedProcessOperator</span><span class="o">&lt;&gt;(</span><span class="n">myProcessFunction</span><span class="o">),</span> <span class="n">x</span> <span class="o">-&gt;</span> <span class="s">&quot;1&quot;</span><span class="o"> [...]
+
+ <span class="n">testHarness</span><span class="o">.</span><span class="na">open</span><span class="o">();</span>
+ <span class="n">testHarness</span><span class="o">.</span><span class="na">processElement</span><span class="o">(</span><span class="s">&quot;world&quot;</span><span class="o">,</span> <span class="mi">10</span><span class="o">);</span>
+ <span class="n">Assert</span><span class="o">.</span><span class="na">assertEquals</span><span class="o">(</span><span class="mi">1</span><span class="o">,</span> <span class="n">testHarness</span><span class="o">.</span><span class="na">numProcessingTimeTimers</span><span class="o&q [...]
+
+ <span class="c1">// Function time is set to 50</span>
+ <span class="n">testHarness</span><span class="o">.</span><span class="na">setProcessingTime</span><span class="o">(</span><span class="mi">50</span><span class="o">);</span>
+ <span class="n">Assert</span><span class="o">.</span><span class="na">assertEquals</span><span class="o">(</span>
+ <span class="n">Lists</span><span class="o">.</span><span class="na">newArrayList</span><span class="o">(</span>
+ <span class="k">new</span> <span class="n">StreamRecord</span><span class="o">&lt;&gt;(</span><span class="s">&quot;hello world&quot;</span><span class="o">,</span> <span class="mi">10</span><span class="o">),</span>
+ <span class="k">new</span> <span class="n">StreamRecord</span><span class="o">&lt;&gt;(</span><span class="s">&quot;Timer triggered at timestamp 50&quot;</span><span class="o">)),</span>
+ <span class="n">testHarness</span><span class="o">.</span><span class="na">extractOutputStreamRecords</span><span class="o">());</span>
+<span class="o">}</span></code></pre></div>
+
+<p>The mechanism to test the multi-input stream operators such as CoProcess functions is similar to the ones described in this article. You should use the TwoInput variant of the harness for these operators, such as <code>TwoInputStreamOperatorTestHarness</code>.</p>
+
+<h1 id="summary">Summary</h1>
+
+<p>In the previous sections we showcased how unit testing in Apache Flink works for stateless, stateful and times-aware-operators. We hope you found the steps easy to follow and execute while developing your Flink applications. If you have any questions or feedback you can reach out to me <a href="https://www.kharekartik.dev/about/">here</a> or contact the community on the <a href="https://flink.apache.org/community.html">Apache Flink user mail [...]
+</description>
+<pubDate>Fri, 07 Feb 2020 13:00:00 +0100</pubDate>
+<link>https://flink.apache.org/news/2020/02/07/a-guide-for-unit-testing-in-apache-flink.html</link>
+<guid isPermaLink="true">/news/2020/02/07/a-guide-for-unit-testing-in-apache-flink.html</guid>
+</item>
+
+<item>
+<title>Apache Flink 1.9.2 Released</title>
+<description><p>The Apache Flink community released the second bugfix version of the Apache Flink 1.9 series.</p>
+
+<p>This release includes 117 fixes and minor improvements for Flink 1.9.1. The list below includes a detailed list of all fixes and improvements.</p>
+
+<p>We highly recommend all users to upgrade to Flink 1.9.2.</p>
+
+<p>Updated Maven dependencies:</p>
+
+<div class="highlight"><pre><code class="language-xml"><span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-java<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.9.2<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span>
+<span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-streaming-java_2.11<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.9.2<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span>
+<span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-clients_2.11<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.9.2<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span></code></pre></div>
+
+<p>You can find the binaries on the updated <a href="/downloads.html">Downloads page</a>.</p>
+
+<p>List of resolved issues:</p>
+
+<h2> Sub-task
+</h2>
+<ul>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12122">FLINK-12122</a>] - Spread out tasks evenly across all available registered TaskManagers
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13360">FLINK-13360</a>] - Add documentation for HBase connector for Table API &amp; SQL
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13361">FLINK-13361</a>] - Add documentation for JDBC connector for Table API &amp; SQL
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13723">FLINK-13723</a>] - Use liquid-c for faster doc generation
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13724">FLINK-13724</a>] - Remove unnecessary whitespace from the docs&#39; sidenav
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13725">FLINK-13725</a>] - Use sassc for faster doc generation
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13726">FLINK-13726</a>] - Build docs with jekyll 4.0.0.pre.beta1
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13791">FLINK-13791</a>] - Speed up sidenav by using group_by
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13817">FLINK-13817</a>] - Expose whether web submissions are enabled
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13818">FLINK-13818</a>] - Check whether web submission are enabled
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14535">FLINK-14535</a>] - Cast exception is thrown when count distinct on decimal fields
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14735">FLINK-14735</a>] - Improve batch schedule check input consumable performance
+</li>
+</ul>
+
+<h2> Bug
+</h2>
+<ul>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10377">FLINK-10377</a>] - Remove precondition in TwoPhaseCommitSinkFunction.notifyCheckpointComplete
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10435">FLINK-10435</a>] - Client sporadically hangs after Ctrl + C
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11120">FLINK-11120</a>] - TIMESTAMPADD function handles TIME incorrectly
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11835">FLINK-11835</a>] - ZooKeeperLeaderElectionITCase.testJobExecutionOnClusterWithLeaderChange failed
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12342">FLINK-12342</a>] - Yarn Resource Manager Acquires Too Many Containers
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12399">FLINK-12399</a>] - FilterableTableSource does not use filters on job run
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13184">FLINK-13184</a>] - Starting a TaskExecutor blocks the YarnResourceManager&#39;s main thread
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13589">FLINK-13589</a>] - DelimitedInputFormat index error on multi-byte delimiters with whole file input splits
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13702">FLINK-13702</a>] - BaseMapSerializerTest.testDuplicate fails on Travis
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13708">FLINK-13708</a>] - Transformations should be cleared because a table environment could execute multiple job
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13740">FLINK-13740</a>] - TableAggregateITCase.testNonkeyedFlatAggregate failed on Travis
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13749">FLINK-13749</a>] - Make Flink client respect classloading policy
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13758">FLINK-13758</a>] - Failed to submit JobGraph when registered hdfs file in DistributedCache
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13799">FLINK-13799</a>] - Web Job Submit Page displays stream of error message when web submit is disables in the config
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13827">FLINK-13827</a>] - Shell variable should be escaped in start-scala-shell.sh
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13862">FLINK-13862</a>] - Update Execution Plan docs
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13945">FLINK-13945</a>] - Instructions for building flink-shaded against vendor repository don&#39;t work
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13969">FLINK-13969</a>] - Resuming Externalized Checkpoint (rocks, incremental, scale down) end-to-end test fails on Travis
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13995">FLINK-13995</a>] - Fix shading of the licence information of netty
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13999">FLINK-13999</a>] - Correct the documentation of MATCH_RECOGNIZE
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14066">FLINK-14066</a>] - Pyflink building failure in master and 1.9.0 version
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14074">FLINK-14074</a>] - MesosResourceManager can&#39;t create new taskmanagers in Session Cluster Mode.
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14175">FLINK-14175</a>] - Upgrade KPL version in flink-connector-kinesis to fix application OOM
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14200">FLINK-14200</a>] - Temporal Table Function Joins do not work on Tables (only TableSources) on the query side
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14235">FLINK-14235</a>] - Kafka010ProducerITCase&gt;KafkaProducerTestBase.testOneToOneAtLeastOnceCustomOperator fails on travis
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14315">FLINK-14315</a>] - NPE with JobMaster.disconnectTaskManager
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14337">FLINK-14337</a>] - HistoryServer does not handle NPE on corruped archives properly
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14347">FLINK-14347</a>] - YARNSessionFIFOITCase.checkForProhibitedLogContents found a log with prohibited string
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14355">FLINK-14355</a>] - Example code in state processor API docs doesn&#39;t compile
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14370">FLINK-14370</a>] - KafkaProducerAtLeastOnceITCase&gt;KafkaProducerTestBase.testOneToOneAtLeastOnceRegularSink fails on Travis
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14382">FLINK-14382</a>] - Incorrect handling of FLINK_PLUGINS_DIR on Yarn
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14398">FLINK-14398</a>] - Further split input unboxing code into separate methods
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14413">FLINK-14413</a>] - Shade-plugin ApacheNoticeResourceTransformer uses platform-dependent encoding
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14434">FLINK-14434</a>] - Dispatcher#createJobManagerRunner should not start JobManagerRunner
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14445">FLINK-14445</a>] - Python module build failed when making sdist
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14447">FLINK-14447</a>] - Network metrics doc table render confusion
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14459">FLINK-14459</a>] - Python module build hangs
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14524">FLINK-14524</a>] - PostgreSQL JDBC sink generates invalid SQL in upsert mode
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14547">FLINK-14547</a>] - UDF cannot be in the join condition in blink planner
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14561">FLINK-14561</a>] - Don&#39;t write FLINK_PLUGINS_DIR ENV variable to Flink configuration
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14562">FLINK-14562</a>] - RMQSource leaves idle consumer after closing
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14574">FLINK-14574</a>] - flink-s3-fs-hadoop doesn&#39;t work with plugins mechanism
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14589">FLINK-14589</a>] - Redundant slot requests with the same AllocationID leads to inconsistent slot table
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14641">FLINK-14641</a>] - Fix description of metric `fullRestarts`
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14673">FLINK-14673</a>] - Shouldn&#39;t expect HMS client to throw NoSuchObjectException for non-existing function
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14683">FLINK-14683</a>] - RemoteStreamEnvironment&#39;s construction function has a wrong method
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14701">FLINK-14701</a>] - Slot leaks if SharedSlotOversubscribedException happens
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14784">FLINK-14784</a>] - CsvTableSink miss delimiter when row start with null member
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14817">FLINK-14817</a>] - &quot;Streaming Aggregation&quot; document contains misleading code examples
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14846">FLINK-14846</a>] - Correct the default writerbuffer size documentation of RocksDB
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14910">FLINK-14910</a>] - DisableAutoGeneratedUIDs fails on keyBy
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14930">FLINK-14930</a>] - OSS Filesystem Uses Wrong Shading Prefix
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14949">FLINK-14949</a>] - Task cancellation can be stuck against out-of-thread error
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14951">FLINK-14951</a>] - State TTL backend end-to-end test fail when taskManager has multiple slot
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14953">FLINK-14953</a>] - Parquet table source should use schema type to build FilterPredicate
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14960">FLINK-14960</a>] - Dependency shading of table modules test fails on Travis
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14976">FLINK-14976</a>] - Cassandra Connector leaks Semaphore on Throwable; hangs on close
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15001">FLINK-15001</a>] - The digest of sub-plan reuse should contain retraction traits for stream physical nodes
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15013">FLINK-15013</a>] - Flink (on YARN) sometimes needs too many slots
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15030">FLINK-15030</a>] - Potential deadlock for bounded blocking ResultPartition.
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15036">FLINK-15036</a>] - Container startup error will be handled out side of the YarnResourceManager&#39;s main thread
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15063">FLINK-15063</a>] - Input group and output group of the task metric are reversed
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15065">FLINK-15065</a>] - RocksDB configurable options doc description error
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15076">FLINK-15076</a>] - Source thread should be interrupted during the Task cancellation
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15234">FLINK-15234</a>] - Hive table created from flink catalog table shouldn&#39;t have null properties in parameters
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15240">FLINK-15240</a>] - is_generic key is missing for Flink table stored in HiveCatalog
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15259">FLINK-15259</a>] - HiveInspector.toInspectors() should convert Flink constant to Hive constant
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15266">FLINK-15266</a>] - NPE in blink planner code gen
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15361">FLINK-15361</a>] - ParquetTableSource should pass predicate in projectFields
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15412">FLINK-15412</a>] - LocalExecutorITCase#testParameterizedTypes failed in travis
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15413">FLINK-15413</a>] - ScalarOperatorsTest failed in travis
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15418">FLINK-15418</a>] - StreamExecMatchRule not set FlinkRelDistribution
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15421">FLINK-15421</a>] - GroupAggsHandler throws java.time.LocalDateTime cannot be cast to java.sql.Timestamp
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15435">FLINK-15435</a>] - ExecutionConfigTests.test_equals_and_hash in pyFlink fails when cpu core numbers is 6
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15443">FLINK-15443</a>] - Use JDBC connector write FLOAT value occur ClassCastException
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15478">FLINK-15478</a>] - FROM_BASE64 code gen type wrong
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15489">FLINK-15489</a>] - WebUI log refresh not working
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15522">FLINK-15522</a>] - Misleading root cause exception when cancelling the job
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15523">FLINK-15523</a>] - ConfigConstants generally excluded from japicmp
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15543">FLINK-15543</a>] - Apache Camel not bundled but listed in flink-dist NOTICE
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15549">FLINK-15549</a>] - Integer overflow in SpillingResettableMutableObjectIterator
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15577">FLINK-15577</a>] - WindowAggregate RelNodes missing Window specs in digest
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15615">FLINK-15615</a>] - Docs: wrong guarantees stated for the file sink
+</li>
+</ul>
+
+<h2> Improvement
+</h2>
+<ul>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11135">FLINK-11135</a>] - Reorder Hadoop config loading in HadoopUtils
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12848">FLINK-12848</a>] - Method equals() in RowTypeInfo should consider fieldsNames
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13729">FLINK-13729</a>] - Update website generation dependencies
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14008">FLINK-14008</a>] - Auto-generate binary licensing
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14104">FLINK-14104</a>] - Bump Jackson to 2.10.1
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14123">FLINK-14123</a>] - Lower the default value of taskmanager.memory.fraction
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14206">FLINK-14206</a>] - Let fullRestart metric count fine grained restarts as well
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14215">FLINK-14215</a>] - Add Docs for TM and JM Environment Variable Setting
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14251">FLINK-14251</a>] - Add FutureUtils#forward utility
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14334">FLINK-14334</a>] - ElasticSearch docs refer to non-existent ExceptionUtils.containsThrowable
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14335">FLINK-14335</a>] - ExampleIntegrationTest in testing docs is incorrect
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14408">FLINK-14408</a>] - In OldPlanner, UDF open method can not be invoke when SQL is optimized
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14557">FLINK-14557</a>] - Clean up the package of py4j
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14639">FLINK-14639</a>] - Metrics User Scope docs refer to wrong class
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14646">FLINK-14646</a>] - Check non-null for key in KeyGroupStreamPartitioner
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14825">FLINK-14825</a>] - Rework state processor api documentation
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14995">FLINK-14995</a>] - Kinesis NOTICE is incorrect
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15113">FLINK-15113</a>] - fs.azure.account.key not hidden from global configuration
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15554">FLINK-15554</a>] - Bump jetty-util-ajax to 9.3.24
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15657">FLINK-15657</a>] - Fix the python table api doc link in Python API tutorial
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15700">FLINK-15700</a>] - Improve Python API Tutorial doc
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15726">FLINK-15726</a>] - Fixing error message in StreamExecTableSourceScan
+</li>
+</ul>
+</description>
+<pubDate>Thu, 30 Jan 2020 13:00:00 +0100</pubDate>
+<link>https://flink.apache.org/news/2020/01/30/release-1.9.2.html</link>
+<guid isPermaLink="true">/news/2020/01/30/release-1.9.2.html</guid>
+</item>
+
+<item>
+<title>State Unlocked: Interacting with State in Apache Flink</title>
+<description><h1 id="introduction">Introduction</h1>
+
+<p>With stateful stream-processing becoming the norm for complex event-driven applications and real-time analytics, <a href="https://flink.apache.org/">Apache Flink</a> is often the backbone for running business logic and managing an organization’s most valuable asset — its data — as application state in Flink.</p>
+
+<p>In order to provide a state-of-the-art experience to Flink developers, the Apache Flink community makes significant efforts to provide the safety and future-proof guarantees organizations need while managing state in Flink. In particular, Flink developers should have sufficient means to access and modify their state, as well as making bootstrapping state with existing data from external systems a piece-of-cake. These efforts span multiple Flink major releases and consist of the [...]
+
+<ol>
+ <li>Evolvable state schema in Apache Flink</li>
+ <li>Flexibility in swapping state backends, and</li>
+ <li>The State processor API, an offline tool to read, write and modify state in Flink</li>
+</ol>
+
+<p>This post discusses the community’s efforts related to state management in Flink, provides some practical examples of how the different features and APIs can be utilized and covers some future ideas for new and improved ways of managing state in Apache Flink.</p>
+
+<h1 id="stream-processing-what-is-state">Stream processing: What is State?</h1>
+
+<p>To set the tone for the remaining of the post, let us first try to explain the very definition of state in stream processing. When it comes to stateful stream processing, state comprises of the information that an application or stream processing engine will remember across events and streams as more realtime (unbounded) and/or offline (bounded) data flow through the system. Most trivial applications are inherently stateful; even the example of a simple COUNT operation, whereby [...]
+
+<p>To better understand how Flink manages state, one can think of Flink like a three-layered state abstraction, as illustrated in the diagram below.</p>
+
+<center>
+<img src="/img/blog/2020-01-29-state-unlocked-interacting-with-state-in-apache-flink/managing-state-in-flink-visual-1.png" width="600px" alt="State in Apache Flink" />
+</center>
+<p><br /></p>
+
+<p>On the top layer, sits the Flink user code, for example, a <code>KeyedProcessFunction</code> that contains some value state. This is a simple variable whose value state annotations makes it automatically fault-tolerant, re-scalable and queryable by the runtime. These variables are backed by the configured state backend that sits either on-heap or on-disk (RocksDB State Backend) and provides data locality, proximity to the computation and speed when it comes to per-re [...]
+
+<p>A savepoint is a snapshot of the distributed, global state of an application at a logical point-in-time and is stored in an external distributed file system or blob storage such as HDFS, or S3. Upon upgrading an application or implementing a code change — such as adding a new operator or changing a field — the Flink job can restart by re-loading the application state from the savepoint into the state backend, making it local and available for the computation and continue proces [...]
+
+<center>
+<img src="/img/blog/2020-01-29-state-unlocked-interacting-with-state-in-apache-flink/managing-state-in-flink-visual-2.png" width="600px" alt="State in Apache Flink" />
+</center>
+<p><br /></p>
+
+<div class="alert alert-info">
+ It is important to remember here that <b>state is one of the most valuable components of a Flink application</b> carrying all the information about both where you are now and where you are going. State is among the most long-lived components in a Flink service since it can be carried across jobs, operators, configurations, new features and bug fixes.
+</div>
+
+<h1 id="schema-evolution-with-apache-flink">Schema Evolution with Apache Flink</h1>
+
+<p>In the previous section, we explained how state is stored and persisted in a Flink application. Let’s now take a look at what happens when evolving state in a stateful Flink streaming application becomes necessary.</p>
+
+<p>Imagine an Apache Flink application that implements a <code>KeyedProcessFunction</code> and contains some <code>ValueState</code>. As illustrated below, within the state descriptor, when registering the type, Flink users specify their <code>TypeInformation</code> that informs Flink about how to serialize the bytes and represents Flink’s internal type system, used to serialize data when shipped across the network or stored in state backends. Fl [...]
+
+<h2 id="state-registration-with-built-in-serialization-in-apache-flink">State registration with built-in serialization in Apache Flink</h2>
+
+<div class="highlight"><pre><code class="language-java"><span class="kd">public</span> <span class="kd">class</span> <span class="nc">MyFunction</span> <span class="kd">extends</span> <span class="n">KeyedProcessFunction</span><span class="o">&lt;</span><span class="n">Key</span><span class=& [...]
+<span class="err"></span>
+ <span class="kd">private</span> <span class="kd">transient</span> <span class="n">ValueState</span><span class="o">&lt;</span><span class="n">MyState</span><span class="o">&gt;</span> <span class="n">valueState</span><span class="o">;</span>
+<span class="err"></span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="nf">open</span><span class="o">(</span><span class="n">Configuration</span> <span class="n">parameters</span><span class="o">)</span> <span class="o">{</span>
+ <span class="n">ValueStateDescriptor</span><span class="o">&lt;</span><span class="n">MyState</span><span class="o">&gt;</span> <span class="n">descriptor</span> <span class="o">=</span>
+ <span class="k">new</span> <span class="n">ValueStateDescriptor</span><span class="o">&lt;&gt;(</span><span class="s">&quot;my-state&quot;</span><span class="o">,</span> <span class="n">TypeInformation</span><span class="o">.</span><span class="na">of</span><span class="o">(< [...]
+<span class="err"></span>
+ <span class="n">valueState</span> <span class="o">=</span> <span class="n">getRuntimeContext</span><span class="o">().</span><span class="na">getState</span><span class="o">(</span><span class="n">descriptor</span><span class="o">);</span>
+ <span class="o">}</span>
+<span class="o">}</span></code></pre></div>
+
+<p>Typically, evolving the schema of an application’s state happens because of some business logic change (adding or dropping fields or changing data types). In all cases, the schema is determined by means of its serializer, and can be thought of in terms of an alter table statement when compared with a database. When a state variable is first introduced it is like running a <code>CREATE_TABLE</code> command, there is a lot of freedom with its execution. However, having [...]
+
+<p><a href="https://flink.apache.org/downloads.html#apache-flink-182">Flink 1.8</a> comes with built-in support for <a href="https://avro.apache.org/">Apache Avro</a> (specifically the <a href="https://avro.apache.org/docs/1.7.7/spec.html">1.7.7 specification</a>) and evolves state schema according to Avro specifications by adding and removing types or even by swapping between generic and specific Avro record types.& [...]
+
+<p>In <a href="https://flink.apache.org/downloads.html#apache-flink-191">Flink 1.9</a> the community added support for schema evolution for POJOs, including the ability to remove existing fields from POJO types or add new fields. The POJO schema evolution tends to be less flexible — when compared to Avro — since it is not possible to change neither the declared field types nor the class name of a POJO type, including its namespace.</p>
+
+<p>With the community’s efforts related to schema evolution, Flink developers can now expect out-of-the-box support for both Avro and POJO formats, with backwards compatibility for all Flink state backends. Future work revolves around adding support for Scala Case Classes, Tuples and other formats. Make sure to subscribe to the <a href="https://flink.apache.org/community.html">Flink mailing list</a> to contribute and stay on top of any upcoming additions in th [...]
+
+<h2 id="peeking-under-the-hood">Peeking Under the Hood</h2>
+
+<p>Now that we have explained how schema evolution in Flink works, let’s describe the challenges of performing schema serialization with Flink under the hood. Flink considers state as a core part of its API stability, in a way that developers should always be able to take a savepoint from one version of Flink and restart it on the next. With schema evolution, every migration needs to be backwards compatible and also compatible with the different state backends. While in the Flink c [...]
+
+<p>For instance, the heap state backend supports lazy serialization and eager deserialization, making the per-record code path always working with Java objects, serializing on a background thread. When restoring, Flink will eagerly deserialize all the data and then start the user code. If a developer plugs in a new serializer, the deserialization happens before Flink ever receives the information.</p>
+
+<p>The RocksDB state backend behaves in the exact opposite manner: it supports eager serialization — because of items being stored on disk and RocksDB only consuming byte arrays. RocksDB provides lazy deserialization simply by downloading files to the local disk, making Flink unaware of what the bytes mean until a serializer is registered.</p>
+
+<p>An additional challenge stems from the fact that different versions of user code contain different classes on their classpath making the serializer used to write into a savepoint likely potentially unavailable at runtime.</p>
+
+<p>To overcome the previously mentioned challenges, we introduced what we call <code>TypeSerializerSnapshot</code>. The <code>TypeSerializerSnapshot</code> stores the configuration of the writer serializer in the snapshot. When restoring it will use that configuration to read back the previous state and check its compatibility with the current version. Using such operation allows Flink to:</p>
+
+<ul>
+ <li>Read the configuration used to write out a snapshot</li>
+ <li>Consume the new user code</li>
+ <li>Check if both items above are compatible</li>
+ <li>Consume the bytes from the snapshot and move forward or alert the user otherwise</li>
+</ul>
+
+<div class="highlight"><pre><code class="language-java"><span class="kd">public</span> <span class="kd">interface</span> <span class="nc">TypeSerializerSnapshot</span><span class="o">&lt;</span><span class="n">T</span><span class="o">&gt;</span> <span class="o">{</span>
+<span class="err"></span>
+ <span class="kt">int</span> <span class="nf">getCurrentVersion</span><span class="o">();</span>
+<span class="err"></span>
+ <span class="kt">void</span> <span class="nf">writeSnapshot</span><span class="o">(</span><span class="n">DataOutputView</span> <span class="n">out</span><span class="o">)</span> <span class="kd">throws</span> <span class="n">IOException</span><span class="o">;</span>
+<span class="err"></span>
+ <span class="kt">void</span> <span class="nf">readSnapshot</span><span class="o">(</span>
+ <span class="kt">int</span> <span class="n">readVersion</span><span class="o">,</span>
+ <span class="n">DataInputView</span> <span class="n">in</span><span class="o">,</span>
+ <span class="n">ClassLoader</span> <span class="n">userCodeClassLoader</span><span class="o">)</span> <span class="kd">throws</span> <span class="n">IOException</span><span class="o">;</span>
+<span class="err"></span>
+ <span class="n">TypeSerializer</span><span class="o">&lt;</span><span class="n">T</span><span class="o">&gt;</span> <span class="nf">restoreSerializer</span><span class="o">();</span>
+<span class="err"></span>
+ <span class="n">TypeSerializerSchemaCompatibility</span><span class="o">&lt;</span><span class="n">T</span><span class="o">&gt;</span> <span class="nf">resolveSchemaCompatibility</span><span class="o">(</span>
+ <span class="n">TypeSerializer</span><span class="o">&lt;</span><span class="n">T</span><span class="o">&gt;</span> <span class="n">newSerializer</span><span class="o">);</span>
+<span class="o">}</span></code></pre></div>
+
+<h2 id="implementing-apache-avro-serialization-in-flink">Implementing Apache Avro Serialization in Flink</h2>
+
+<p>Apache Avro is a data serialization format that has very well-defined schema migration semantics and supports both reader and writer schemas. During normal Flink execution the reader and writer schemas will be the same. However, when upgrading an application they may be different and with schema evolution, Flink will be able to migrate objects with their schemas.</p>
+
+<div class="highlight"><pre><code class="language-java"><span class="kd">public</span> <span class="kd">class</span> <span class="nc">AvroSerializerSnapshot</span><span class="o">&lt;</span><span class="n">T</span><span class="o">&gt;</span> <span class="kd">implements</span> <span class= [...]
+ <span class="kd">private</span> <span class="n">Schema</span> <span class="n">runtimeSchema</span><span class="o">;</span>
+ <span class="kd">private</span> <span class="n">Schema</span> <span class="n">previousSchema</span><span class="o">;</span>
+<span class="err"></span>
+ <span class="nd">@SuppressWarnings</span><span class="o">(</span><span class="s">&quot;WeakerAccess&quot;</span><span class="o">)</span>
+ <span class="kd">public</span> <span class="nf">AvroSerializerSnapshot</span><span class="o">()</span> <span class="o">{</span> <span class="o">}</span>
+<span class="err"></span>
+ <span class="n">AvroSerializerSnapshot</span><span class="o">(</span><span class="n">Schema</span> <span class="n">schema</span><span class="o">)</span> <span class="o">{</span>
+ <span class="k">this</span><span class="o">.</span><span class="na">runtimeSchema</span> <span class="o">=</span> <span class="n">schema</span><span class="o">;</span>
+ <span class="o">}</span></code></pre></div>
+
+<p>This is a sketch of our Avro serializer. It uses the provided schemas and delegates to Apache Avro for all (de)-serialization. Let’s take a look at one possible implementation of a <code>TypeSerializerSnapshot</code> that supports schema migration for Avro.</p>
+
+<h1 id="writing-out-the-snapshot">Writing out the snapshot</h1>
+
+<p>When serializing out the snapshot, the snapshot configuration will write two pieces of information; the current snapshot configuration version and the serializer configuration.</p>
+
+<div class="highlight"><pre><code class="language-java"> <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="kt">int</span> <span class="nf">getCurrentVersion</span><span class="o">()</span> <span class="o">{</span>
+ <span class="k">return</span> <span class="mi">1</span><span class="o">;</span>
+ <span class="o">}</span>
+<span class="err"></span>
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="nf">writeSnapshot</span><span class="o">(</span><span class="n">DataOutputView</span> <span class="n">out</span><span class="o">)</span> <span class="kd">throws</span> <span class="n">IOException</span> <span class=& [...]
+ <span class="n">out</span><span class="o">.</span><span class="na">writeUTF</span><span class="o">(</span><span class="n">runtimeSchema</span><span class="o">.</span><span class="na">toString</span><span class="o">(</span><span class="kc">false</span><span class="o">));< [...]
+ <span class="o">}</span></code></pre></div>
+
+<p>The version is used to version the snapshot configuration object itself while the <code>writeSnapshot</code> method writes out all the information we need to understand the current format; the runtime schema.</p>
+
+<div class="highlight"><pre><code class="language-java"> <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="nf">readSnapshot</span><span class="o">(</span>
+ <span class="kt">int</span> <span class="n">readVersion</span><span class="o">,</span>
+ <span class="n">DataInputView</span> <span class="n">in</span><span class="o">,</span>
+ <span class="n">ClassLoader</span> <span class="n">userCodeClassLoader</span><span class="o">)</span> <span class="kd">throws</span> <span class="n">IOException</span> <span class="o">{</span>
+
+ <span class="k">assert</span> <span class="n">readVersion</span> <span class="o">==</span> <span class="mi">1</span><span class="o">;</span>
+ <span class="kd">final</span> <span class="n">String</span> <span class="n">previousSchemaDefinition</span> <span class="o">=</span> <span class="n">in</span><span class="o">.</span><span class="na">readUTF</span><span class="o">();</span>
+ <span class="k">this</span><span class="o">.</span><span class="na">previousSchema</span> <span class="o">=</span> <span class="n">parseAvroSchema</span><span class="o">(</span><span class="n">previousSchemaDefinition</span><span class="o">);</span>
+ <span class="k">this</span><span class="o">.</span><span class="na">runtimeType</span> <span class="o">=</span> <span class="n">findClassOrFallbackToGeneric</span><span class="o">(</span>
+ <span class="n">userCodeClassLoader</span><span class="o">,</span>
+ <span class="n">previousSchema</span><span class="o">.</span><span class="na">getFullName</span><span class="o">());</span>
+<span class="err"></span>
+ <span class="k">this</span><span class="o">.</span><span class="na">runtimeSchema</span> <span class="o">=</span> <span class="n">tryExtractAvroSchema</span><span class="o">(</span><span class="n">userCodeClassLoader</span><span class="o">,</span> <span class="n">runtimeType</span><span [...]
+ <span class="o">}</span></code></pre></div>
+<p>Now when Flink restores it is able to read back in the writer schema used to serialize the data. The current runtime schema is discovered on the class path using some Java reflection magic.</p>
+
+<p>Once we have both of these we can compare them for compatibility. Perhaps nothing has changed and the schemas are compatible as is.</p>
+
+<div class="highlight"><pre><code class="language-java"> <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="n">TypeSerializerSchemaCompatibility</span><span class="o">&lt;</span><span class="n">T</span><span class="o">&gt;</span> <span class="nf">resolveSchemaCompatibility</span><span class="o">(</span>
+ <span class="n">TypeSerializer</span><span class="o">&lt;</span><span class="n">T</span><span class="o">&gt;</span> <span class="n">newSerializer</span><span class="o">)</span> <span class="o">{</span>
+<span class="err"></span>
+ <span class="k">if</span> <span class="o">(!(</span><span class="n">newSerializer</span> <span class="k">instanceof</span> <span class="n">AvroSerializer</span><span class="o">))</span> <span class="o">{</span>
+ <span class="k">return</span> <span class="n">TypeSerializerSchemaCompatibility</span><span class="o">.</span><span class="na">incompatible</span><span class="o">();</span>
+ <span class="o">}</span>
+<span class="err"></span>
+ <span class="k">if</span> <span class="o">(</span><span class="n">Objects</span><span class="o">.</span><span class="na">equals</span><span class="o">(</span><span class="n">previousSchema</span><span class="o">,</span> <span class="n">runtimeSchema</span><span class="o">)) [...]
+ <span class="k">return</span> <span class="n">TypeSerializerSchemaCompatibility</span><span class="o">.</span><span class="na">compatibleAsIs</span><span class="o">();</span>
+ <span class="o">}</span></code></pre></div>
+
+<p>Otherwise, the schemas are compared using Avro’s compatibility checks and they may either be compatible with a migration or incompatible.</p>
+
+<div class="highlight"><pre><code class="language-java"> <span class="kd">final</span> <span class="n">SchemaPairCompatibility</span> <span class="n">compatibility</span> <span class="o">=</span> <span class="n">SchemaCompatibility</span>
+ <span class="o">.</span><span class="na">checkReaderWriterCompatibility</span><span class="o">(</span><span class="n">previousSchema</span><span class="o">,</span> <span class="n">runtimeSchema</span><span class="o">);</span>
+<span class="err"></span>
+ <span class="k">return</span> <span class="nf">avroCompatibilityToFlinkCompatibility</span><span class="o">(</span><span class="n">compatibility</span><span class="o">);</span>
+ <span class="o">}</span></code></pre></div>
+
+<p>If they are compatible with migration then Flink will restore a new serializer that can read the old schema and deserialize into the new runtime type which is in effect a migration.</p>
+
+<div class="highlight"><pre><code class="language-java"> <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="n">TypeSerializer</span><span class="o">&lt;</span><span class="n">T</span><span class="o">&gt;</span> <span class="nf">restoreSerializer</span><span class="o">()</span> <span class="o">{</span>
+ <span class="k">if</span> <span class="o">(</span><span class="n">previousSchema</span> <span class="o">!=</span> <span class="kc">null</span><span class="o">)</span> <span class="o">{</span>
+ <span class="k">return</span> <span class="k">new</span> <span class="n">AvroSerializer</span><span class="o">&lt;&gt;(</span><span class="n">runtimeType</span><span class="o">,</span> <span class="n">runtimeSchema</span><span class="o">,</span> <span class="n">previousSchema</span& [...]
+ <span class="o">}</span> <span class="k">else</span> <span class="o">{</span>
+ <span class="k">return</span> <span class="k">new</span> <span class="n">AvroSerializer</span><span class="o">&lt;&gt;(</span><span class="n">runtimeType</span><span class="o">,</span> <span class="n">runtimeSchema</span><span class="o">,</span> <span class="n">runtimeSchema</span&g [...]
+ <span class="o">}</span>
+ <span class="o">}</span>
+<span class="o">}</span></code></pre></div>
+
+<h1 id="the-state-processor-api-reading-writing-and-modifying-flink-state">The State Processor API: Reading, writing and modifying Flink state</h1>
+
+<p>The State Processor API allows reading from and writing to Flink savepoints. Some of the interesting use cases it can be used for are:</p>
+
+<ul>
+ <li>Analyzing state for interesting patterns</li>
+ <li>Troubleshooting or auditing jobs by checking for state discrepancies</li>
+ <li>Bootstrapping state for new applications</li>
+ <li>Modifying savepoints such as:
+ <ul>
+ <li>Changing the maximum parallelism of a savepoint after deploying a Flink job</li>
+ <li>Introducing breaking schema updates to a Flink application</li>
+ <li>Correcting invalid state in a Flink savepoint</li>
+ </ul>
+ </li>
+</ul>
+
+<p>In a <a href="https://flink.apache.org/feature/2019/09/13/state-processor-api.html">previous blog post</a>, we discussed the State Processor API in detail, the community’s motivation behind introducing the feature in Flink 1.9, what you can use the API for and how you can use it. Essentially, the State Processor API is based around a relational model of mapping your Flink job state to a database, as illustrated in the diagram below. We encourage you to < [...]
+
+<ul>
+ <li>Reading Keyed and Operator State with the State Processor API and</li>
+ <li>Writing and Bootstrapping Keyed and Operator State with the State Processor API</li>
+</ul>
+
+<p>Stay tuned for more details and guidance around this feature of Flink.</p>
+
+<center>
+<img src="/img/blog/2020-01-29-state-unlocked-interacting-with-state-in-apache-flink/managing-state-in-flink-state-processor-api-visual-1.png" width="600px" alt="State Processor API in Apache Flink" />
+</center>
+<p><br /></p>
+
+<center>
+<img src="/img/blog/2020-01-29-state-unlocked-interacting-with-state-in-apache-flink/managing-state-in-flink-state-processor-api-visual-2.png" width="600px" alt="State Processor API in Apache Flink" />
+</center>
+<p><br /></p>
+
+<h1 id="looking-ahead-more-ways-to-interact-with-state-in-flink">Looking ahead: More ways to interact with State in Flink</h1>
+
+<p>There is a lot of discussion happening in the community related to extending the way Flink developers interact with state in their Flink applications. Regarding the State Processor API, some thoughts revolve around further broadening the API’s scope beyond its current ability to read from and write to both keyed and operator state. In upcoming releases, the State processor API will be extended to support both reading from and writing to windows and have a first-class integration [...]
+
+<p>Beyond widening the scope of the State Processor API, the Flink community is discussing a few additional ways to improve the way developers interact with state in Flink. One of them is the proposal for a Unified Savepoint Format (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-41%3A+Unify+Binary+format+for+Keyed+State">FLIP-41</a>) for all keyed state backends. Such improvement aims at introducing a unified binary format across all savepoint [...]
+
+<p>The community is also discussing the ability to have upgradability dry runs in upcoming Flink releases. Having such functionality in Flink allows developers to detect incompatible updates offline without the need of starting a new Flink job from scratch. For example, Flink users will be able to uncover topology or schema incompatibilities upon upgrading a Flink job, without having to load the state back to a running Flink job in the first place. Additionally, with upgradability [...]
+
+<p>With all the exciting new functionality added in Flink 1.9 as well as some solid ideas and discussions around bringing state in Flink to the next level, the community is committed to making state in Apache Flink a fundamental element of the framework, something that is ever-present across versions and upgrades of your application and a component that is a true first-class citizen in Apache Flink. We encourage you to sign up to the <a href="https://flink.apache.org/commu [...]
+</description>
+<pubDate>Wed, 29 Jan 2020 13:00:00 +0100</pubDate>
+<link>https://flink.apache.org/news/2020/01/29/state-unlocked-interacting-with-state-in-apache-flink.html</link>
+<guid isPermaLink="true">/news/2020/01/29/state-unlocked-interacting-with-state-in-apache-flink.html</guid>
+</item>
+
+<item>
+<title>Advanced Flink Application Patterns Vol.1: Case Study of a Fraud Detection System</title>
+<description><p>In this series of blog posts you will learn about three powerful Flink patterns for building streaming applications:</p>
+
+<ul>
+ <li>Dynamic updates of application logic</li>
+ <li>Dynamic data partitioning (shuffle), controlled at runtime</li>
+ <li>Low latency alerting based on custom windowing logic (without using the window API)</li>
+</ul>
+
+<p>These patterns expand the possibilities of what is achievable with statically defined data flows and provide the building blocks to fulfill complex business requirements.</p>
+
+<p><strong>Dynamic updates of application logic</strong> allow Flink jobs to change at runtime, without downtime from stopping and resubmitting the code.<br />
+<br />
+<strong>Dynamic data partitioning</strong> provides the ability to change how events are distributed and grouped by Flink at runtime. Such functionality often becomes a natural requirement when building jobs with dynamically reconfigurable application logic.<br />
+<br />
+<strong>Custom window management</strong> demonstrates how you can utilize the low level <a href="https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/process_function.html">process function API</a>, when the native <a href="https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/windows.html">window API</a> is not exactly matching your requirements. Specifically, you will learn how to impl [...]
+
+<p>These patterns build on top of core Flink functionality, however, they might not be immediately apparent from the framework’s documentation as explaining and presenting the motivation behind them is not always trivial without a concrete use case. That is why we will showcase these patterns with a practical example that offers a real-world usage scenario for Apache Flink — a <em>Fraud Detection</em> engine.
+We hope that this series will place these powerful approaches into your tool belt and enable you to take on new and exciting tasks.</p>
+
+<p>In the first blog post of the series we will look at the high-level architecture of the demo application, describe its components and their interactions. We will then deep dive into the implementation details of the first pattern in the series - <strong>dynamic data partitioning</strong>.</p>
+
+<p>You will be able to run the full Fraud Detection Demo application locally and look into the details of the implementation by using the accompanying GitHub repository.</p>
+
+<h3 id="fraud-detection-demo">Fraud Detection Demo</h3>
+
+<p>The full source code for our fraud detection demo is open source and available online. To run it locally, check out the following repository and follow the steps in the README:</p>
+
+<p><a href="https://github.com/afedulov/fraud-detection-demo">https://github.com/afedulov/fraud-detection-demo</a></p>
+
+<p>You will see the demo is a self-contained application - it only requires <code>docker</code> and <code>docker-compose</code> to be built from sources and includes the following components:</p>
+
+<ul>
+ <li>Apache Kafka (message broker) with ZooKeeper</li>
+ <li>Apache Flink (<a href="https://ci.apache.org/projects/flink/flink-docs-stable/concepts/glossary.html#flink-application-cluster">application cluster</a>)</li>
+ <li>Fraud Detection Web App</li>
+</ul>
+
+<p>The high-level goal of the Fraud Detection engine is to consume a stream of financial transactions and evaluate them against a set of rules. These rules are subject to frequent changes and tweaks. In a real production system, it is important to be able to add and remove them at runtime, without incurring an expensive penalty of stopping and restarting the job.</p>
+
+<p>When you navigate to the demo URL in your browser, you will be presented with the following UI:</p>
+
+<center>
+ <img src="/img/blog/2019-11-19-demo-fraud-detection/ui.png" width="800px" alt="Figure 1: Demo UI" />
+ <br />
+ <i><small>Figure 1: Fraud Detection Demo UI</small></i>
+ </center>
+<p><br /></p>
+
+<p>On the left side, you can see a visual representation of financial transactions flowing through the system after you click the “Start” button. The slider at the top allows you to control the number of generated transactions per second. The middle section is devoted to managing the rules evaluated by Flink. From here, you can create new rules as well as issue control commands, such as clearing Flink’s state.</p>
+
+<p>The demo out-of-the-box comes with a set of predefined sample rules. You can click the <em>Start</em> button and, after some time, will observe alerts displayed in the right section of the UI. These alerts are the result of Flink evaluating the generated transactions stream against the predefined rules.</p>
+
+<p>Our sample fraud detection system consists of three main components:</p>
+
+<ol>
+ <li>Frontend (React)</li>
+ <li>Backend (SpringBoot)</li>
+ <li>Fraud Detection application (Apache Flink)</li>
+</ol>
+
+<p>Interactions between the main elements are depicted in <em>Figure 2</em>.</p>
+
+<center>
+ <img src="/img/blog/2019-11-19-demo-fraud-detection/architecture.png" width="800px" alt="Figure 2: Demo Components" />
+ <br />
+ <i><small>Figure 2: Fraud Detection Demo Components</small></i>
+ </center>
+<p><br /></p>
+
+<p>The Backend exposes a REST API to the Frontend for creating/deleting rules as well as issuing control commands for managing the demo execution. It then relays those Frontend actions to Flink by sending them via a “Control” Kafka topic. The Backend additionally includes a <em>Transactions Generator</em> component, which sends an emulated stream of money transfer events to Flink via a separate “Transactions” topic. Alerts generated by Flink are consumed by the Backend [...]
+
+<p>Now that you are familiar with the overall layout and the goal of our Fraud Detection engine, let’s now go into the details of what is required to implement such a system.</p>
+
+<h3 id="dynamic-data-partitioning">Dynamic Data Partitioning</h3>
+
+<p>The first pattern we will look into is Dynamic Data Partitioning.</p>
+
+<p>If you have used Flink’s DataStream API in the past, you are undoubtedly familiar with the <strong>keyBy</strong> method. Keying a stream shuffles all the records such that elements with the same key are assigned to the same partition. This means all records with the same key are processed by the same physical instance of the next operator.</p>
+
+<p>In a typical streaming application, the choice of key is fixed, determined by some static field within the elements. For instance, when building a simple window-based aggregation of a stream of transactions, we might always group by the transactions account id.</p>
+
+<div class="highlight"><pre><code class="language-java"><span class="n">DataStream</span><span class="o">&lt;</span><span class="n">Transaction</span><span class="o">&gt;</span> <span class="n">input</span> <span class="o">=</span> <span class="c1">// [...]</span>
+<span class="n">DataStream</span><span class="o">&lt;...&gt;</span> <span class="n">windowed</span> <span class="o">=</span> <span class="n">input</span>
+ <span class="o">.</span><span class="na">keyBy</span><span class="o">(</span><span class="nl">Transaction:</span><span class="o">:</span><span class="n">getAccountId</span><span class="o">)</span>
+ <span class="o">.</span><span class="na">window</span><span class="o">(</span><span class="cm">/*window specification*/</span><span class="o">);</span></code></pre></div>
+
+<p>This approach is the main building block for achieving horizontal scalability in a wide range of use cases. However, in the case of an application striving to provide flexibility in business logic at runtime, this is not enough.
+To understand why this is the case, let us start with articulating a realistic sample rule definition for our fraud detection system in the form of a functional requirement:</p>
+
+<p><em>“Whenever the <strong>sum</strong> of the accumulated <strong>payment amount</strong> from the same <strong>payer</strong> to the same <strong>beneficiary</strong> within the <strong>duration of a week</strong> is <strong>greater</strong> than <strong>1 000 000 $</strong> - fire an alert.”</em></p>
+
+<p>In this formulation we can spot a number of parameters that we would like to be able to specify in a newly-submitted rule and possibly even later modify or tweak at runtime:</p>
+
+<ol>
+ <li>Aggregation field (payment amount)</li>
+ <li>Grouping fields (payer + beneficiary)</li>
+ <li>Aggregation function (sum)</li>
+ <li>Window duration (1 week)</li>
+ <li>Limit (1 000 000)</li>
+ <li>Limit operator (greater)</li>
+</ol>
+
+<p>Accordingly, we will use the following simple JSON format to define the aforementioned parameters:</p>
+
+<div class="highlight"><pre><code class="language-json"><span class="p">{</span>
+ <span class="nt">&quot;ruleId&quot;</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
+ <span class="nt">&quot;ruleState&quot;</span><span class="p">:</span> <span class="s2">&quot;ACTIVE&quot;</span><span class="p">,</span>
+ <span class="nt">&quot;groupingKeyNames&quot;</span><span class="p">:</span> <span class="p">[</span><span class="s2">&quot;payerId&quot;</span><span class="p">,</span> <span class="s2">&quot;beneficiaryId&quot;</span><span class="p">],</span>
+ <span class="nt">&quot;aggregateFieldName&quot;</span><span class="p">:</span> <span class="s2">&quot;paymentAmount&quot;</span><span class="p">,</span>
+ <span class="nt">&quot;aggregatorFunctionType&quot;</span><span class="p">:</span> <span class="s2">&quot;SUM&quot;</span><span class="p">,</span>
+ <span class="nt">&quot;limitOperatorType&quot;</span><span class="p">:</span> <span class="s2">&quot;GREATER&quot;</span><span class="p">,</span>
+ <span class="nt">&quot;limit&quot;</span><span class="p">:</span> <span class="mi">1000000</span><span class="p">,</span>
+ <span class="nt">&quot;windowMinutes&quot;</span><span class="p">:</span> <span class="mi">10080</span>
+<span class="p">}</span></code></pre></div>
+
+<p>At this point, it is important to understand that <strong><code>groupingKeyNames</code></strong> determine the actual physical grouping of events - all Transactions with the same values of specified parameters (e.g. <em>payer #25 -&gt; beneficiary #12</em>) have to be aggregated in the same physical instance of the evaluating operator. Naturally, the process of distributing data in such a way in Flink’s API is realised by a <code>key [...]
+
+<p>Most examples in Flink’s <code>keyBy()</code><a href="https://ci.apache.org/projects/flink/flink-docs-stable/dev/api_concepts.html#define-keys-using-field-expressions">documentation</a> use a hard-coded <code>KeySelector</code>, which extracts specific fixed events’ fields. However, to support the desired flexibility, we have to extract them in a more dynamic fashion based on the specifications of the rules. For this, we will have to [...]
+
+<p>On a high level, our main processing pipeline looks like this:</p>
+
+<div class="highlight"><pre><code class="language-java"><span class="n">DataStream</span><span class="o">&lt;</span><span class="n">Alert</span><span class="o">&gt;</span> <span class="n">alerts</span> <span class="o">=</span>
+ <span class="n">transactions</span>
+ <span class="o">.</span><span class="na">process</span><span class="o">(</span><span class="k">new</span> <span class="nf">DynamicKeyFunction</span><span class="o">())</span>
+ <span class="o">.</span><span class="na">keyBy</span><span class="o">(</span><span class="cm">/* some key selector */</span><span class="o">);</span>
+ <span class="o">.</span><span class="na">process</span><span class="o">(</span><span class="cm">/* actual calculations and alerting */</span><span class="o">)</span></code></pre></div>
+
+<p>We have previously established that each rule defines a <strong><code>groupingKeyNames</code></strong> parameter that specifies which combination of fields will be used for the incoming events’ grouping. Each rule might use an arbitrary combination of these fields. At the same time, every incoming event potentially needs to be evaluated against multiple rules. This implies that events might simultaneously need to be present at multiple parallel instances [...]
+
+<center>
+<img src="/img/blog/2019-11-19-demo-fraud-detection/shuffle_function_1.png" width="800px" alt="Figure 3: Forking events with Dynamic Key Function" />
+<br />
+<i><small>Figure 3: Forking events with Dynamic Key Function</small></i>
+</center>
+<p><br /></p>
+
+<p><code>DynamicKeyFunction</code> iterates over a set of defined rules and prepares every event to be processed by a <code>keyBy()</code> function by extracting the required grouping keys:</p>
+
+<div class="highlight"><pre><code class="language-java"><span class="kd">public</span> <span class="kd">class</span> <span class="nc">DynamicKeyFunction</span>
+ <span class="kd">extends</span> <span class="n">ProcessFunction</span><span class="o">&lt;</span><span class="n">Transaction</span><span class="o">,</span> <span class="n">Keyed</span><span class="o">&lt;</span><span class="n">Transaction</span><span class="o">,</span> <span cla [...]
+ <span class="o">...</span>
+ <span class="cm">/* Simplified */</span>
+ <span class="n">List</span><span class="o">&lt;</span><span class="n">Rule</span><span class="o">&gt;</span> <span class="n">rules</span> <span class="o">=</span> <span class="cm">/* Rules that are initialized somehow.</span>
+<span class="cm"> Details will be discussed in a future blog post. */</span><span class="o">;</span>
+
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="nf">processElement</span><span class="o">(</span>
+ <span class="n">Transaction</span> <span class="n">event</span><span class="o">,</span>
+ <span class="n">Context</span> <span class="n">ctx</span><span class="o">,</span>
+ <span class="n">Collector</span><span class="o">&lt;</span><span class="n">Keyed</span><span class="o">&lt;</span><span class="n">Transaction</span><span class="o">,</span> <span class="n">String</span><span class="o">,</span> <span class="n">Integer</span><span class="o& [...]
+
+ <span class="k">for</span> <span class="o">(</span><span class="n">Rule</span> <span class="n">rule</span> <span class="o">:</span><span class="n">rules</span><span class="o">)</span> <span class="o">{</span>
+ <span class="n">out</span><span class="o">.</span><span class="na">collect</span><span class="o">(</span>
+ <span class="k">new</span> <span class="n">Keyed</span><span class="o">&lt;&gt;(</span>
+ <span class="n">event</span><span class="o">,</span>
+ <span class="n">KeysExtractor</span><span class="o">.</span><span class="na">getKey</span><span class="o">(</span><span class="n">rule</span><span class="o">.</span><span class="na">getGroupingKeyNames</span><span class="o">(),</span> <span class="n">event</span><span class=& [...]
+ <span class="n">rule</span><span class="o">.</span><span class="na">getRuleId</span><span class="o">()));</span>
+ <span class="o">}</span>
+ <span class="o">}</span>
+ <span class="o">...</span>
+<span class="o">}</span></code></pre></div>
+<p><code>KeysExtractor.getKey()</code> uses reflection to extract the required values of <code>groupingKeyNames</code> fields from events and combines them as a single concatenated String key, e.g <code>"{payerId=25;beneficiaryId=12}"</code>. Flink will calculate the hash of this key and assign the processing of this particular combination to a specific server in the cluster. This will allow tracking all transactions between <em>p [...]
+
+<p>Notice that a wrapper class <code>Keyed</code> with the following signature was introduced as the output type of <code>DynamicKeyFunction</code>:</p>
+
+<div class="highlight"><pre><code class="language-java"><span class="kd">public</span> <span class="kd">class</span> <span class="nc">Keyed</span><span class="o">&lt;</span><span class="n">IN</span><span class="o">,</span> <span class="n">KEY</span><span class="o">,</span> [...]
+ <span class="kd">private</span> <span class="n">IN</span> <span class="n">wrapped</span><span class="o">;</span>
+ <span class="kd">private</span> <span class="n">KEY</span> <span class="n">key</span><span class="o">;</span>
+ <span class="kd">private</span> <span class="n">ID</span> <span class="n">id</span><span class="o">;</span>
+
+ <span class="o">...</span>
+ <span class="kd">public</span> <span class="n">KEY</span> <span class="nf">getKey</span><span class="o">(){</span>
+ <span class="k">return</span> <span class="n">key</span><span class="o">;</span>
+ <span class="o">}</span>
+<span class="o">}</span></code></pre></div>
+
+<p>Fields of this POJO carry the following information: <code>wrapped</code> is the original transaction event, <code>key</code> is the result of using <code>KeysExtractor</code> and <code>id</code> is the ID of the Rule that caused the dispatch of the event (according to the rule-specific grouping logic).</p>
+
+<p>Events of this type will be the input to the <code>keyBy()</code> function in the main processing pipeline and allow the use of a simple lambda-expression as a <a href="https://ci.apache.org/projects/flink/flink-docs-stable/dev/api_concepts.html#define-keys-using-key-selector-functions"><code>KeySelector</code></a> for the final step of implementing dynamic data shuffle.</p>
+
+<div class="highlight"><pre><code class="language-java"><span class="n">DataStream</span><span class="o">&lt;</span><span class="n">Alert</span><span class="o">&gt;</span> <span class="n">alerts</span> <span class="o">=</span>
+ <span class="n">transactions</span>
+ <span class="o">.</span><span class="na">process</span><span class="o">(</span><span class="k">new</span> <span class="nf">DynamicKeyFunction</span><span class="o">())</span>
+ <span class="o">.</span><span class="na">keyBy</span><span class="o">((</span><span class="n">keyed</span><span class="o">)</span> <span class="o">-&gt;</span> <span class="n">keyed</span><span class="o">.</span><span class="na">getKey</span><span class="o">());< [...]
+ <span class="o">.</span><span class="na">process</span><span class="o">(</span><span class="k">new</span> <span class="nf">DynamicAlertFunction</span><span class="o">())</span></code></pre></div>
+
+<p>By applying <code>DynamicKeyFunction</code> we are implicitly copying events for performing parallel per-rule evaluation within a Flink cluster. By doing so, we achieve an important property - horizontal scalability of rules’ processing. Our system will be capable of handling more rules by adding more servers to the cluster, i.e. increasing the parallelism. This property is achieved at the cost of data duplication, which might become an issue depending on the specifi [...]
+
+<h3 id="summary">Summary:</h3>
+
+<p>In this blog post, we have discussed the motivation behind supporting dynamic, runtime changes to a Flink application by looking at a sample use case - a Fraud Detection engine. We have described the overall architecture and interactions between its components as well as provided references for building and running a demo Fraud Detection application in a dockerized setup. We then showed the details of implementing a <strong>dynamic data partitioning pattern</strong> [...]
+
+<p>To remain focused on describing the core mechanics of the pattern, we kept the complexity of the DSL and the underlying rules engine to a minimum. Going forward, it is easy to imagine adding extensions such as allowing more sophisticated rule definitions, including filtering of certain events, logical rules chaining, and other more advanced functionality.</p>
+
+<p>In the second part of this series, we will describe how the rules make their way into the running Fraud Detection engine. Additionally, we will go over the implementation details of the main processing function of the pipeline - <em>DynamicAlertFunction()</em>.</p>
+
+<center>
+<img src="/img/blog/2019-11-19-demo-fraud-detection/end-to-end.png" width="800px" alt="Figure 4: End-to-end pipeline" />
+<br />
+<i><small>Figure 4: End-to-end pipeline</small></i>
+</center>
+<p><br /></p>
+
+<p>In the next article, we will see how Flink’s broadcast streams can be utilized to help steer the processing within the Fraud Detection engine at runtime (Dynamic Application Updates pattern).</p>
+</description>
+<pubDate>Wed, 15 Jan 2020 13:00:00 +0100</pubDate>
+<link>https://flink.apache.org/news/2020/01/15/demo-fraud-detection.html</link>
+<guid isPermaLink="true">/news/2020/01/15/demo-fraud-detection.html</guid>
+</item>
+
+<item>
+<title>Apache Flink 1.8.3 Released</title>
+<description><p>The Apache Flink community released the third bugfix version of the Apache Flink 1.8 series.</p>
+
+<p>This release includes 45 fixes and minor improvements for Flink 1.8.2. The list below includes a detailed list of all fixes and improvements.</p>
+
+<p>We highly recommend all users to upgrade to Flink 1.8.3.</p>
+
+<p>Updated Maven dependencies:</p>
+
+<div class="highlight"><pre><code class="language-xml"><span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-java<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.8.3<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span>
+<span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-streaming-java_2.11<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.8.3<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span>
+<span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-clients_2.11<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.8.3<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span></code></pre></div>
+
+<p>You can find the binaries on the updated <a href="/downloads.html">Downloads page</a>.</p>
+
+<p>List of resolved issues:</p>
+
+<h2> Sub-task
+</h2>
+<ul>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13723">FLINK-13723</a>] - Use liquid-c for faster doc generation
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13724">FLINK-13724</a>] - Remove unnecessary whitespace from the docs&#39; sidenav
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13725">FLINK-13725</a>] - Use sassc for faster doc generation
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13726">FLINK-13726</a>] - Build docs with jekyll 4.0.0.pre.beta1
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13791">FLINK-13791</a>] - Speed up sidenav by using group_by
+</li>
+</ul>
+
+<h2> Bug
+</h2>
+<ul>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12342">FLINK-12342</a>] - Yarn Resource Manager Acquires Too Many Containers
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13184">FLINK-13184</a>] - Starting a TaskExecutor blocks the YarnResourceManager&#39;s main thread
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13728">FLINK-13728</a>] - Fix wrong closing tag order in sidenav
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13746">FLINK-13746</a>] - Elasticsearch (v2.3.5) sink end-to-end test fails on Travis
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13749">FLINK-13749</a>] - Make Flink client respect classloading policy
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13892">FLINK-13892</a>] - HistoryServerTest failed on Travis
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13936">FLINK-13936</a>] - NOTICE-binary is outdated
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13966">FLINK-13966</a>] - Jar sorting in collect_license_files.sh is locale dependent
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13995">FLINK-13995</a>] - Fix shading of the licence information of netty
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13999">FLINK-13999</a>] - Correct the documentation of MATCH_RECOGNIZE
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14009">FLINK-14009</a>] - Cron jobs broken due to verifying incorrect NOTICE-binary file
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14010">FLINK-14010</a>] - Dispatcher &amp; JobManagers don&#39;t give up leadership when AM is shut down
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14043">FLINK-14043</a>] - SavepointMigrationTestBase is super slow
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14107">FLINK-14107</a>] - Kinesis consumer record emitter deadlock under event time alignment
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14175">FLINK-14175</a>] - Upgrade KPL version in flink-connector-kinesis to fix application OOM
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14235">FLINK-14235</a>] - Kafka010ProducerITCase&gt;KafkaProducerTestBase.testOneToOneAtLeastOnceCustomOperator fails on travis
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14315">FLINK-14315</a>] - NPE with JobMaster.disconnectTaskManager
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14337">FLINK-14337</a>] - HistoryServerTest.testHistoryServerIntegration failed on Travis
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14347">FLINK-14347</a>] - YARNSessionFIFOITCase.checkForProhibitedLogContents found a log with prohibited string
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14370">FLINK-14370</a>] - KafkaProducerAtLeastOnceITCase&gt;KafkaProducerTestBase.testOneToOneAtLeastOnceRegularSink fails on Travis
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14398">FLINK-14398</a>] - Further split input unboxing code into separate methods
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14413">FLINK-14413</a>] - shade-plugin ApacheNoticeResourceTransformer uses platform-dependent encoding
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14434">FLINK-14434</a>] - Dispatcher#createJobManagerRunner should not start JobManagerRunner
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14562">FLINK-14562</a>] - RMQSource leaves idle consumer after closing
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14589">FLINK-14589</a>] - Redundant slot requests with the same AllocationID leads to inconsistent slot table
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-15036">FLINK-15036</a>] - Container startup error will be handled out side of the YarnResourceManager&#39;s main thread
+</li>
+</ul>
+
+<h2> Improvement
+</h2>
+<ul>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12848">FLINK-12848</a>] - Method equals() in RowTypeInfo should consider fieldsNames
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13729">FLINK-13729</a>] - Update website generation dependencies
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13965">FLINK-13965</a>] - Keep hasDeprecatedKeys and deprecatedKeys methods in ConfigOption and mark it with @Deprecated annotation
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13967">FLINK-13967</a>] - Generate full binary licensing via collect_license_files.sh
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13968">FLINK-13968</a>] - Add travis check for the correctness of the binary licensing
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13991">FLINK-13991</a>] - Add git exclusion for 1.9+ features to 1.8
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14008">FLINK-14008</a>] - Auto-generate binary licensing
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14104">FLINK-14104</a>] - Bump Jackson to 2.10.1
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14123">FLINK-14123</a>] - Lower the default value of taskmanager.memory.fraction
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14215">FLINK-14215</a>] - Add Docs for TM and JM Environment Variable Setting
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14334">FLINK-14334</a>] - ElasticSearch docs refer to non-existent ExceptionUtils.containsThrowable
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14639">FLINK-14639</a>] - Fix the document of Metrics that has an error for `User Scope`
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14646">FLINK-14646</a>] - Check non-null for key in KeyGroupStreamPartitioner
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14995">FLINK-14995</a>] - Kinesis NOTICE is incorrect
+</li>
+</ul>
+</description>
+<pubDate>Wed, 11 Dec 2019 13:00:00 +0100</pubDate>
+<link>https://flink.apache.org/news/2019/12/11/release-1.8.3.html</link>
+<guid isPermaLink="true">/news/2019/12/11/release-1.8.3.html</guid>
+</item>
+
+<item>
+<title>Running Apache Flink on Kubernetes with KUDO</title>
+<description><p>A common use case for Apache Flink is streaming data analytics together with Apache Kafka, which provides a pub/sub model and durability for data streams. To achieve elastic scalability, both are typically deployed in clustered environments, and increasingly on top of container orchestration platforms like Kubernetes. The <a href="https://kubernetes.io/docs/concepts/extend-kubernetes/operator/">Operator pattern</a> provides an extension mechani [...]
+
+<p>In this blog post we demonstrate how to orchestrate a streaming data analytics application based on Flink and Kafka with KUDO. It consists of a Flink job that checks financial transactions for fraud, and two microservices that generate and display the transactions. You can find more details about this demo in the <a href="https://github.com/kudobuilder/operators/tree/master/repository/flink/docs/demo/financial-fraud">KUDO Operators repository</a>, including [...]
+
+<p style="display: block; text-align: center; margin-top: 20px; margin-bottom: 20px">
+ <img src="/img/blog/2019-11-06-flink-kubernetes-kudo/flink-kudo-architecture.png" width="600px" alt="Application: My App" />
+</p>
+
+<h2 id="prerequisites">Prerequisites</h2>
+
+<p>You can run this demo on your local machine using <a href="https://github.com/kubernetes/minikube">minikube</a>. The instructions below were tested with minikube v1.5.1 and Kubernetes v1.16.2 but should work on any Kubernetes version above v1.15.0. First, start a minikube cluster with enough capacity:</p>
+
+<p><code>minikube start --cpus=6 --memory=9216 --disk-size=10g</code></p>
+
+<p>If you’re using a different way to provision Kubernetes, make sure you have at least 6 CPU Cores, 9 GB of RAM and 10 GB of disk space available.</p>
+
+<p>Install the <code>kubectl</code> CLI tool. The KUDO CLI is a plugin for the Kubernetes CLI. The official instructions for installing and setting up kubectl are <a href="https://kubernetes.io/docs/tasks/tools/install-kubectl/">here</a>.</p>
+
+<p>Next, let’s install the KUDO CLI. At the time of this writing, the latest KUDO version is v0.10.0. You can find the CLI binaries for download <a href="https://github.com/kudobuilder/kudo/releases">here</a>. Download the <code>kubectl-kudo</code> binary for your OS and architecture.</p>
+
+<p>If you’re using Homebrew on MacOS, you can install the CLI via:</p>
+
+<div class="highlight"><pre><code>$ brew tap kudobuilder/tap
+$ brew install kudo-cli
+</code></pre></div>
+
+<p>Now, let’s initialize KUDO on our Kubernetes cluster:</p>
+
+<div class="highlight"><pre><code>$ kubectl kudo init
+$KUDO_HOME has been configured at /Users/gerred/.kudo
+</code></pre></div>
+
+<p>This will create several resources. First, it will create the <a href="https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/">Custom Resource Definitions</a>, <a href="https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/">service account</a>, and <a href="https://kubernetes.io/docs/reference/access-authn-authz/rbac/">role bindings</a> necessary for KUD [...]
+
+<p>The KUDO CLI leverages the kubectl plugin system, which gives you all its functionality under <code>kubectl kudo</code>. This is a convenient way to install and deal with your KUDO Operators. For our demo, we use Kafka and Flink which depend on ZooKeeper. To make the ZooKeeper Operator available on the cluster, run:</p>
+
+<div class="highlight"><pre><code>$ kubectl kudo install zookeeper --version=0.3.0 --skip-instance
+</code></pre></div>
+
+<p>The –skip-instance flag skips the creation of a ZooKeeper instance. The flink-demo Operator that we’re going to install below will create it as a dependency instead. Now let’s make the Kafka and Flink Operators available the same way:</p>
+
+<div class="highlight"><pre><code>$ kubectl kudo install kafka --version=1.2.0 --skip-instance
+</code></pre></div>
+
+<div class="highlight"><pre><code>$ kubectl kudo install flink --version=0.2.1 --skip-instance
+</code></pre></div>
+
+<p>This installs all the Operator versions needed for our demo.</p>
+
+<h2 id="financial-fraud-demo">Financial Fraud Demo</h2>
+
+<p>In our financial fraud demo we have two microservices, called “generator” and “actor”. The generator produces transactions with random amounts and writes them into a Kafka topic. Occasionally, the value will be over 10,000 which is considered fraud for the purpose of this demo. The Flink job subscribes to the Kafka topic and detects fraudulent transactions. When it does, it submits them to another Kafka topic which the actor consumes. The actor simply displays each fraudulent tr [...]
+
+<p>The KUDO CLI by default installs Operators from the <a href="https://github.com/kudobuilder/operators/">official repository</a>, but it also supports installation from your local filesystem. This is useful if you want to develop your own Operator, or modify this demo for your own purposes.</p>
+
+<p>First, clone the “kudobuilder/operators” repository via:</p>
+
+<div class="highlight"><pre><code>$ git clone https://github.com/kudobuilder/operators.git
+</code></pre></div>
+
+<p>Next, change into the “operators” directory and install the demo-operator from your local filesystem:</p>
+
+<div class="highlight"><pre><code>$ cd operators
+$ kubectl kudo install repository/flink/docs/demo/financial-fraud/demo-operator --instance flink-demo
+instance.kudo.dev/v1beta1/flink-demo created
+</code></pre></div>
+
+<p>This time we didn’t include the –skip-instance flag, so KUDO will actually deploy all the components, including Flink, Kafka, and ZooKeeper. KUDO orchestrates deployments and other lifecycle operations using <a href="https://kudo.dev/docs/concepts.html#plan">plans</a> that were defined by the Operator developer. Plans are similar to <a href="https://en.wikipedia.org/wiki/Runbook">runbooks</a> and encapsulate all the procedures required [...]
+
+<div class="highlight"><pre><code>$ kubectl kudo plan status --instance flink-demo
+Plan(s) for "flink-demo" in namespace "default":
+.
+└── flink-demo (Operator-Version: "flink-demo-0.1.4" Active-Plan: "deploy")
+ └── Plan deploy (serial strategy) [IN_PROGRESS]
+ ├── Phase dependencies [IN_PROGRESS]
+ │ ├── Step zookeeper (COMPLETE)
+ │ └── Step kafka (IN_PROGRESS)
+ ├── Phase flink-cluster [PENDING]
+ │ └── Step flink (PENDING)
+ ├── Phase demo [PENDING]
+ │ ├── Step gen (PENDING)
+ │ └── Step act (PENDING)
+ └── Phase flink-job [PENDING]
+ └── Step submit (PENDING)
+</code></pre></div>
+
+<p>The output shows that the “deploy” plan is in progress and that it consists of 4 phases: “dependencies”, “flink-cluster”, “demo” and “flink-job”. The “dependencies” phase includes steps for “zookeeper” and “kafka”. This is where both dependencies get installed, before KUDO continues to install the Flink cluster and the demo itself. We also see that ZooKeeper installation completed, and that Kafka installation is currently in progress. We can view details about Kafka’s deployment [...]
+
+<div class="highlight"><pre><code>$ kubectl kudo plan status --instance flink-demo-kafka
+Plan(s) for "flink-demo-kafka" in namespace "default":
+.
+└── flink-demo-kafka (Operator-Version: "kafka-1.2.0" Active-Plan: "deploy")
+ ├── Plan deploy (serial strategy) [IN_PROGRESS]
+ │ └── Phase deploy-kafka [IN_PROGRESS]
+ │ └── Step deploy (IN_PROGRESS)
+ └── Plan not-allowed (serial strategy) [NOT ACTIVE]
+ └── Phase not-allowed (serial strategy) [NOT ACTIVE]
+ └── Step not-allowed (serial strategy) [NOT ACTIVE]
+ └── not-allowed [NOT ACTIVE]
+</code></pre></div>
+
+<p>After Kafka was successfully installed the next phase “flink-cluster” will start and bring up, you guessed it, your flink-cluster. After this is done, the demo phase creates the generator and actor pods that generate and display transactions for this demo. Lastly, we have the flink-job phase in which we submit the actual FinancialFraudJob to the Flink cluster. Once the flink job is submitted, we will be able to see fraud logs in our actor pod shortly after.</p>
+
+<p>After a while, the state of all plans, phases and steps will change to “COMPLETE”. Now we can view the Flink dashboard to verify that our job is running. To access it from outside the Kubernetes cluster, first start the client proxy, then open the URL below in your browser:</p>
+
+<div class="highlight"><pre><code>$ kubectl proxy
+</code></pre></div>
+
+<p><a href="http://127.0.0.1:8001/api/v1/namespaces/default/services/flink-demo-flink-jobmanager:ui/proxy/#/overview">http://127.0.0.1:8001/api/v1/namespaces/default/services/flink-demo-flink-jobmanager:ui/proxy/#/overview</a></p>
+
+<p>It should look similar to this, depending on your local machine and how many cores you have available:</p>
+
+<p style="display: block; text-align: center; margin-top: 20px; margin-bottom: 20px">
+ <img src="/img/blog/2019-11-06-flink-kubernetes-kudo/flink-dashboard-ui.png" width="600px" alt="Application: My App" />
+</p>
+
+<p>The job is up and running and we should now be able to see fraudulent transaction in the logs of the actor pod:</p>
+
+<div class="highlight"><pre><code>$ kubectl logs $(kubectl get pod -l actor=flink-demo -o jsonpath="{.items[0].metadata.name}")
+Broker: flink-demo-kafka-kafka-0.flink-demo-kafka-svc:9093
+Topic: fraud
+
+Detected Fraud: TransactionAggregate {startTimestamp=0, endTimestamp=1563395831000, totalAmount=19895:
+Transaction{timestamp=1563395778000, origin=1, target='3', amount=8341}
+Transaction{timestamp=1563395813000, origin=1, target='3', amount=8592}
+Transaction{timestamp=1563395817000, origin=1, target='3', amount=2802}
+Transaction{timestamp=1563395831000, origin=1, target='3', amount=160}}
+</code></pre></div>
+
+<p>If you add the “-f” flag to the previous command, you can follow along as more transactions are streaming in and are evaluated by our Flink job.</p>
+
+<h2 id="conclusion">Conclusion</h2>
+
+<p>In this blog post we demonstrated how to easily deploy an end-to-end streaming data application on Kubernetes using KUDO. We deployed a Flink job and two microservices, as well as all the required infrastructure - Flink, Kafka, and ZooKeeper using just a few kubectl commands. To find out more about KUDO, visit the <a href="https://kudo.dev">project website</a> or join the community on <a href="https://kubernetes.slack.com/messages/kudo/">Sla [...]
+</description>
+<pubDate>Mon, 09 Dec 2019 13:00:00 +0100</pubDate>
+<link>https://flink.apache.org/news/2019/12/09/flink-kubernetes-kudo.html</link>
+<guid isPermaLink="true">/news/2019/12/09/flink-kubernetes-kudo.html</guid>
+</item>
+
+<item>
+<title>How to query Pulsar Streams using Apache Flink</title>
+<description><p>In a previous <a href="https://flink.apache.org/2019/05/03/pulsar-flink.html">story</a> on the Flink blog, we explained the different ways that <a href="https://flink.apache.org/">Apache Flink</a> and <a href="https://pulsar.apache.org/">Apache Pulsar</a> can integrate to provide elastic data processing at large scale. This blog post discusses the new developments and integrations between the two fra [...]
+
+<h1 id="a-short-intro-to-apache-pulsar">A short intro to Apache Pulsar</h1>
+
+<p>Apache Pulsar is a flexible pub/sub messaging system, backed by durable log storage. Some of the framework’s highlights include multi-tenancy, a unified message model, structured event streams and a cloud-native architecture that make it a perfect fit for a wide set of use cases, ranging from billing, payments and trading services all the way to the unification of the different messaging architectures in an organization. If you are interested in finding out more about Pulsar, yo [...]
+
+<h1 id="existing-pulsar--flink-integration-apache-flink-16">Existing Pulsar &amp; Flink integration (Apache Flink 1.6+)</h1>
+
+<p>The existing integration between Pulsar and Flink exploits Pulsar as a message queue in a Flink application. Flink developers can utilize Pulsar as a streaming source and streaming sink for their Flink applications by selecting a specific Pulsar source and connecting to their desired Pulsar cluster and topic:</p>
+
+<div class="highlight"><pre><code class="language-java"><span class="c1">// create and configure Pulsar consumer</span>
+<span class="n">PulsarSourceBuilder</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span><span class="n">builder</span> <span class="o">=</span> <span class="n">PulsarSourceBuilder</span>
+ <span class="o">.</span><span class="na">builder</span><span class="o">(</span><span class="k">new</span> <span class="nf">SimpleStringSchema</span><span class="o">())</span>
+ <span class="o">.</span><span class="na">serviceUrl</span><span class="o">(</span><span class="n">serviceUrl</span><span class="o">)</span>
+ <span class="o">.</span><span class="na">topic</span><span class="o">(</span><span class="n">inputTopic</span><span class="o">)</span>
+ <span class="o">.</span><span class="na">subsciptionName</span><span class="o">(</span><span class="n">subscription</span><span class="o">);</span>
+<span class="n">SourceFunction</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">src</span> <span class="o">=</span> <span class="n">builder</span><span class="o">.</span><span class="na">build</span><span class="o"&g [...]
+<span class="c1">// ingest DataStream with Pulsar consumer</span>
+<span class="n">DataStream</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">words</span> <span class="o">=</span> <span class="n">env</span><span class="o">.</span><span class="na">addSource</span><span class="o"> [...]
+
+<p>Pulsar streams can then get connected to the Flink processing logic…</p>
+
+<div class="highlight"><pre><code class="language-java"><span class="c1">// perform computation on DataStream (here a simple WordCount)</span>
+<span class="n">DataStream</span><span class="o">&lt;</span><span class="n">WordWithCount</span><span class="o">&gt;</span> <span class="n">wc</span> <span class="o">=</span> <span class="n">words</span>
+ <span class="o">.</span><span class="na">flatmap</span><span class="o">((</span><span class="n">FlatMapFunction</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">WordWithCount</span><span class="o">&gt;)</span> <span class=" [...]
+ <span class="n">collector</span><span class="o">.</span><span class="na">collect</span><span class="o">(</span><span class="k">new</span> <span class="nf">WordWithCount</span><span class="o">(</span><span class="n">word</span><span class="o">,</span> <span class="mi">1</ [...]
+ <span class="o">})</span>
+
+ <span class="o">.</span><span class="na">returns</span><span class="o">(</span><span class="n">WordWithCount</span><span class="o">.</span><span class="na">class</span><span class="o">)</span>
+ <span class="o">.</span><span class="na">keyBy</span><span class="o">(</span><span class="s">&quot;word&quot;</span><span class="o">)</span>
+ <span class="o">.</span><span class="na">timeWindow</span><span class="o">(</span><span class="n">Time</span><span class="o">.</span><span class="na">seconds</span><span class="o">(</span><span class="mi">5</span><span class="o">))</span>
+ <span class="o">.</span><span class="na">reduce</span><span class="o">((</span><span class="n">ReduceFunction</span><span class="o">&lt;</span><span class="n">WordWithCount</span><span class="o">&gt;)</span> <span class="o">(</span><span class="n">c1</span><span class="o" [...]
+ <span class="k">new</span> <span class="nf">WordWithCount</span><span class="o">(</span><span class="n">c1</span><span class="o">.</span><span class="na">word</span><span class="o">,</span> <span class="n">c1</span><span class="o">.</span><span class="na">count</span> [...]
+
+<p>…and then get emitted back to Pulsar (used now as a sink), sending one’s computation results downstream, back to a Pulsar topic:</p>
+
+<div class="highlight"><pre><code class="language-java"><span class="c1">// emit result via Pulsar producer </span>
+<span class="n">wc</span><span class="o">.</span><span class="na">addSink</span><span class="o">(</span><span class="k">new</span> <span class="n">FlinkPulsarProducer</span><span class="o">&lt;&gt;(</span>
+ <span class="n">serviceUrl</span><span class="o">,</span>
+ <span class="n">outputTopic</span><span class="o">,</span>
+ <span class="k">new</span> <span class="nf">AuthentificationDisabled</span><span class="o">(),</span>
+ <span class="n">wordWithCount</span> <span class="o">-&gt;</span> <span class="n">wordWithCount</span><span class="o">.</span><span class="na">toString</span><span class="o">().</span><span class="na">getBytes</span><span class="o">(</span><span class="n">UTF_8</span><span class=" [...]
+ <span class="n">wordWithCount</span> <span class="o">-&gt;</span> <span class="n">wordWithCount</span><span class="o">.</span><span class="na">word</span><span class="o">)</span>
+<span class="o">);</span></code></pre></div>
+
+<p>Although this is a great first integration step, the existing design is not leveraging the full power of Pulsar. Some shortcomings of the integration with Flink 1.6.0 relate to Pulsar neither being utilized as durable storage nor having schema integration with Flink, resulting in manual input when describing an application’s schema registry.</p>
+
+<h1 id="pulsars-integration-with-flink-19-using-pulsar-as-a-flink-catalog">Pulsar’s integration with Flink 1.9: Using Pulsar as a Flink catalog</h1>
+
+<p>The latest integration between <a href="https://flink.apache.org/downloads.html#apache-flink-191">Flink 1.9.0</a> and Pulsar addresses most of the previously mentioned shortcomings. The <a href="https://flink.apache.org/news/2019/02/13/unified-batch-streaming-blink.html">contribution of Alibaba’s Blink to the Flink repository</a> adds many enhancements and new features to the processing framework that make the integration with Pulsar s [...]
+
+<h1 id="leveraging-the-flink--pulsar-schema-integration">Leveraging the Flink &lt;&gt; Pulsar Schema Integration</h1>
+
+<p>Before delving into the integration details and how you can use Pulsar schema with Flink, let us describe how schema in Pulsar works. Schema in Apache Pulsar already co-exists and serves as the representation of the data on the broker side of the framework, something that makes schema registry with external systems obsolete. Additionally, the data schema in Pulsar is associated with each topic so both producers and consumers send data with predefined schema information, while th [...]
+
+<p>Below you can find an example of Pulsar’s schema on both the producer and consumer side. On the producer side, you can specify which schema you want to use and Pulsar then sends a POJO class without the need to perform any serialization/deserialization. Similarly, on the consumer end, you can also specify the data schema and upon receiving the data, Pulsar will automatically validate the schema information, fetch the schema of the given version and then deserialize the data back [...]
+
+<div class="highlight"><pre><code class="language-java"><span class="c1">// Create producer with Struct schema and send messages</span>
+<span class="n">Producer</span><span class="o">&lt;</span><span class="n">User</span><span class="o">&gt;</span> <span class="n">producer</span> <span class="o">=</span> <span class="n">client</span><span class="o">.</span><span class="na">newProducer</span><span class="o" [...]
+<span class="n">producer</span><span class="o">.</span><span class="na">newMessage</span><span class="o">()</span>
+ <span class="o">.</span><span class="na">value</span><span class="o">(</span><span class="n">User</span><span class="o">.</span><span class="na">builder</span><span class="o">()</span>
+ <span class="o">.</span><span class="na">userName</span><span class="o">(</span><span class="err">“</span><span class="n">pulsar</span><span class="o">-</span><span class="n">user</span><span class="err">”</span><span class="o">)</span>
+ <span class="o">.</span><span class="na">userId</span><span class="o">(</span><span class="mi">1L</span><span class="o">)</span>
+ <span class="o">.</span><span class="na">build</span><span class="o">())</span>
+ <span class="o">.</span><span class="na">send</span><span class="o">();</span></code></pre></div>
+
+<div class="highlight"><pre><code class="language-java"><span class="c1">// Create consumer with Struct schema and receive messages</span>
+<span class="n">Consumer</span><span class="o">&lt;</span><span class="n">User</span><span class="o">&gt;</span> <span class="n">consumer</span> <span class="o">=</span> <span class="n">client</span><span class="o">.</span><span class="na">newCOnsumer</span><span class="o" [...]
+<span class="n">consumer</span><span class="o">.</span><span class="na">receive</span><span class="o">();</span></code></pre></div>
+
+<p>Let’s assume we have an application that specifies a schema to the producer and/or consumer. Upon receiving the schema information, the producer (or consumer) — that is connected to the broker — will transfer such information so that the broker can then perform schema registration, validations and schema compatibility checks before returning or rejecting the schema as illustrated in the diagram below:</p>
+
+<center>
+<img src="/img/blog/flink-pulsar-sql-blog-post-visual.png" width="600px" alt="Pulsar Schema" />
+</center>
+<p><br /></p>
+
+<p>Not only is Pulsar able to handle and store the schema information, but is additionally able to handle any schema evolution — where necessary. Pulsar will effectively manage any schema evolution in the broker, keeping track of all different versions of your schema while performing any necessary compatibility checks.</p>
+
+<p>Moreover, when messages are published on the producer side, Pulsar will tag each message with the schema version as part of each message’s metadata. On the consumer side, when the message is received and the metadata is deserialized, Pulsar will check the schema version associated with this message and will fetch the corresponding schema information from the broker. As a result, when Pulsar integrates with a Flink application it uses the pre-existing schema information and maps [...]
+
+<p>For the cases when Flink users do not interact with schema directly or make use of primitive schema (for example, using a topic to store a string or long number), Pulsar will either convert the message payload into a Flink row, called ‘value’ or — for the cases of structured schema types, like JSON and AVRO — Pulsar will extract the individual fields from the schema information and will map the fields to Flink’s type system. Finally, all metadata information associated with eac [...]
+
+<center>
+<img src="/img/blog/flink-pulsar-sql-blog-post-visual-primitive-avro-schema.png" width="600px" alt="Primitive and AVRO Schema" />
+</center>
+<p><br /></p>
+
+<p>Once all the schema information is mapped to Flink’s type system, you can start building a Pulsar source, sink or catalog in Flink based on the specified schema information as illustrated below:</p>
+
+<h1 id="flink--pulsar-read-data-from-pulsar">Flink &amp; Pulsar: Read data from Pulsar</h1>
+
+<ul>
+ <li>Create a Pulsar source for streaming queries</li>
+</ul>
+
+<div class="highlight"><pre><code class="language-java"><span class="n">val</span> <span class="n">env</span> <span class="o">=</span> <span class="n">StreamExecutionEnvironment</span><span class="o">.</span><span class="na">getExecutionEnvironment</span>
+<span class="n">val</span> <span class="n">props</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">Properties</span><span class="o">()</span>
+<span class="n">props</span><span class="o">.</span><span class="na">setProperty</span><span class="o">(</span><span class="s">&quot;service.url&quot;</span><span class="o">,</span> <span class="s">&quot;pulsar://...&quot;</span><span class="o">)</span>
+<span class="n">props</span><span class="o">.</span><span class="na">setProperty</span><span class="o">(</span><span class="s">&quot;admin.url&quot;</span><span class="o">,</span> <span class="s">&quot;http://...&quot;</span><span class="o">)</span>
+<span class="n">props</span><span class="o">.</span><span class="na">setProperty</span><span class="o">(</span><span class="s">&quot;partitionDiscoveryIntervalMillis&quot;</span><span class="o">,</span> <span class="s">&quot;5000&quot;</span><span class="o">)</span>
+<span class="n">props</span><span class="o">.</span><span class="na">setProperty</span><span class="o">(</span><span class="s">&quot;startingOffsets&quot;</span><span class="o">,</span> <span class="s">&quot;earliest&quot;</span><span class="o">)</span>
+<span class="n">props</span><span class="o">.</span><span class="na">setProperty</span><span class="o">(</span><span class="s">&quot;topic&quot;</span><span class="o">,</span> <span class="s">&quot;test-source-topic&quot;</span><span class="o">)</span>
+<span class="n">val</span> <span class="n">source</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">FlinkPulsarSource</span><span class="o">(</span><span class="n">props</span><span class="o">)</span>
+<span class="c1">// you don&#39;t need to provide a type information to addSource since FlinkPulsarSource is ResultTypeQueryable</span>
+<span class="n">val</span> <span class="n">dataStream</span> <span class="o">=</span> <span class="n">env</span><span class="o">.</span><span class="na">addSource</span><span class="o">(</span><span class="n">source</span><span class="o">)(</span><span class="kc">null</span& [...]
+
+<span class="c1">// chain operations on dataStream of Row and sink the output</span>
+<span class="c1">// end method chaining</span>
+
+<span class="n">env</span><span class="o">.</span><span class="na">execute</span><span class="o">()</span></code></pre></div>
+
+<ul>
+ <li>Register topics in Pulsar as streaming tables</li>
+</ul>
+
+<div class="highlight"><pre><code class="language-java"><span class="n">val</span> <span class="n">env</span> <span class="o">=</span> <span class="n">StreamExecutionEnvironment</span><span class="o">.</span><span class="na">getExecutionEnvironment</span>
+<span class="n">val</span> <span class="n">tEnv</span> <span class="o">=</span> <span class="n">StreamTableEnvironment</span><span class="o">.</span><span class="na">create</span><span class="o">(</span><span class="n">env</span><span class="o">)</span>
+
+<span class="n">val</span> <span class="n">prop</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">Properties</span><span class="o">()</span>
+<span class="n">prop</span><span class="o">.</span><span class="na">setProperty</span><span class="o">(</span><span class="s">&quot;service.url&quot;</span><span class="o">,</span> <span class="n">serviceUrl</span><span class="o">)</span>
+<span class="n">prop</span><span class="o">.</span><span class="na">setProperty</span><span class="o">(</span><span class="s">&quot;admin.url&quot;</span><span class="o">,</span> <span class="n">adminUrl</span><span class="o">)</span>
+<span class="n">prop</span><span class="o">.</span><span class="na">setProperty</span><span class="o">(</span><span class="s">&quot;flushOnCheckpoint&quot;</span><span class="o">,</span> <span class="s">&quot;true&quot;</span><span class="o">)</span>
+<span class="n">prop</span><span class="o">.</span><span class="na">setProperty</span><span class="o">(</span><span class="s">&quot;failOnWrite&quot;</span><span class="o">,</span> <span class="s">&quot;true&quot;</span><span class="o">)</span>
+<span class="n">props</span><span class="o">.</span><span class="na">setProperty</span><span class="o">(</span><span class="s">&quot;topic&quot;</span><span class="o">,</span> <span class="s">&quot;test-sink-topic&quot;</span><span class="o">)</span>
+
+<span class="n">tEnv</span>
+ <span class="o">.</span><span class="na">connect</span><span class="o">(</span><span class="k">new</span> <span class="nf">Pulsar</span><span class="o">().</span><span class="na">properties</span><span class="o">(</span><span class="n">props</span><span class="o">))</span>
+ <span class="o">.</span><span class="na">inAppendMode</span><span class="o">()</span>
+ <span class="o">.</span><span class="na">registerTableSource</span><span class="o">(</span><span class="s">&quot;sink-table&quot;</span><span class="o">)</span>
+
+<span class="n">val</span> <span class="n">sql</span> <span class="o">=</span> <span class="s">&quot;INSERT INTO sink-table .....&quot;</span>
+<span class="n">tEnv</span><span class="o">.</span><span class="na">sqlUpdate</span><span class="o">(</span><span class="n">sql</span><span class="o">)</span>
+<span class="n">env</span><span class="o">.</span><span class="na">execute</span><span class="o">()</span></code></pre></div>
+
+<h1 id="flink--pulsar-write-data-to-pulsar">Flink &amp; Pulsar: Write data to Pulsar</h1>
+
+<ul>
+ <li>Create a Pulsar sink for streaming queries</li>
+</ul>
+
+<div class="highlight"><pre><code class="language-java"><span class="n">val</span> <span class="n">env</span> <span class="o">=</span> <span class="n">StreamExecutionEnvironment</span><span class="o">.</span><span class="na">getExecutionEnvironment</span>
+<span class="n">val</span> <span class="n">stream</span> <span class="o">=</span> <span class="o">.....</span>
+
+<span class="n">val</span> <span class="n">prop</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">Properties</span><span class="o">()</span>
+<span class="n">prop</span><span class="o">.</span><span class="na">setProperty</span><span class="o">(</span><span class="s">&quot;service.url&quot;</span><span class="o">,</span> <span class="n">serviceUrl</span><span class="o">)</span>
+<span class="n">prop</span><span class="o">.</span><span class="na">setProperty</span><span class="o">(</span><span class="s">&quot;admin.url&quot;</span><span class="o">,</span> <span class="n">adminUrl</span><span class="o">)</span>
+<span class="n">prop</span><span class="o">.</span><span class="na">setProperty</span><span class="o">(</span><span class="s">&quot;flushOnCheckpoint&quot;</span><span class="o">,</span> <span class="s">&quot;true&quot;</span><span class="o">)</span>
+<span class="n">prop</span><span class="o">.</span><span class="na">setProperty</span><span class="o">(</span><span class="s">&quot;failOnWrite&quot;</span><span class="o">,</span> <span class="s">&quot;true&quot;</span><span class="o">)</span>
+<span class="n">props</span><span class="o">.</span><span class="na">setProperty</span><span class="o">(</span><span class="s">&quot;topic&quot;</span><span class="o">,</span> <span class="s">&quot;test-sink-topic&quot;</span><span class="o">)</span>
+
+<span class="n">stream</span><span class="o">.</span><span class="na">addSink</span><span class="o">(</span><span class="k">new</span> <span class="nf">FlinkPulsarSink</span><span class="o">(</span><span class="n">prop</span><span class="o">,</span> <span class="n">DummyTopicKe [...]
+<span class="n">env</span><span class="o">.</span><span class="na">execute</span><span class="o">()</span></code></pre></div>
+
+<ul>
+ <li>Write a streaming table to Pulsar</li>
+</ul>
+
+<div class="highlight"><pre><code class="language-java"><span class="n">val</span> <span class="n">env</span> <span class="o">=</span> <span class="n">StreamExecutionEnvironment</span><span class="o">.</span><span class="na">getExecutionEnvironment</span>
+<span class="n">val</span> <span class="n">tEnv</span> <span class="o">=</span> <span class="n">StreamTableEnvironment</span><span class="o">.</span><span class="na">create</span><span class="o">(</span><span class="n">env</span><span class="o">)</span>
+
+<span class="n">val</span> <span class="n">prop</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">Properties</span><span class="o">()</span>
+<span class="n">prop</span><span class="o">.</span><span class="na">setProperty</span><span class="o">(</span><span class="s">&quot;service.url&quot;</span><span class="o">,</span> <span class="n">serviceUrl</span><span class="o">)</span>
+<span class="n">prop</span><span class="o">.</span><span class="na">setProperty</span><span class="o">(</span><span class="s">&quot;admin.url&quot;</span><span class="o">,</span> <span class="n">adminUrl</span><span class="o">)</span>
+<span class="n">prop</span><span class="o">.</span><span class="na">setProperty</span><span class="o">(</span><span class="s">&quot;flushOnCheckpoint&quot;</span><span class="o">,</span> <span class="s">&quot;true&quot;</span><span class="o">)</span>
+<span class="n">prop</span><span class="o">.</span><span class="na">setProperty</span><span class="o">(</span><span class="s">&quot;failOnWrite&quot;</span><span class="o">,</span> <span class="s">&quot;true&quot;</span><span class="o">)</span>
+<span class="n">props</span><span class="o">.</span><span class="na">setProperty</span><span class="o">(</span><span class="s">&quot;topic&quot;</span><span class="o">,</span> <span class="s">&quot;test-sink-topic&quot;</span><span class="o">)</span>
+
+<span class="n">tEnv</span>
+ <span class="o">.</span><span class="na">connect</span><span class="o">(</span><span class="k">new</span> <span class="nf">Pulsar</span><span class="o">().</span><span class="na">properties</span><span class="o">(</span><span class="n">props</span><span class="o">))</span>
+ <span class="o">.</span><span class="na">inAppendMode</span><span class="o">()</span>
+ <span class="o">.</span><span class="na">registerTableSource</span><span class="o">(</span><span class="s">&quot;sink-table&quot;</span><span class="o">)</span>
+
+<span class="n">val</span> <span class="n">sql</span> <span class="o">=</span> <span class="s">&quot;INSERT INTO sink-table .....&quot;</span>
+<span class="n">tEnv</span><span class="o">.</span><span class="na">sqlUpdate</span><span class="o">(</span><span class="n">sql</span><span class="o">)</span>
+<span class="n">env</span><span class="o">.</span><span class="na">execute</span><span class="o">()</span></code></pre></div>
+
+<p>In every instance, Flink developers only need to specify the properties of how Flink will connect to a Pulsar cluster without worrying about any schema registry, or serialization/deserialization actions and register the Pulsar cluster as a source, sink or streaming table in Flink. Once all three elements are put together, Pulsar can then be registered as a catalog in Flink, something that drastically simplifies how you process and query data like, for example, writing a program [...]
+
+<h1 id="next-steps--future-integration">Next Steps &amp; Future Integration</h1>
+
+<p>The goal of the integration between Pulsar and Flink is to simplify how developers use the two frameworks to build a unified data processing stack. As we progress from the classical Lamda architectures — where an online, speeding layer is combined with an offline, batch layer to run data computations — Flink and Pulsar present a great combination in providing a truly unified data processing stack. We see Flink as a unified computation engine, handling both online (streaming) and [...]
+
+<p>There is still a lot of ongoing work and effort from both communities in getting the integration even better, such as a new source API (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface">FLIP-27</a>) that will allow the <a href="http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Discussion-Flink-Pulsar-Connector-td22019.html">contribution of the Pulsar connectors to the Flink communit [...]
+
+<p>You can find a more detailed overview of the integration work between the two communities in this <a href="https://youtu.be/3sBXXfgl5vs">recording video</a> from Flink Forward Europe 2019 or sign up to the <a href="https://flink.apache.org/community.html#mailing-lists">Flink dev mailing list</a> for the latest contribution and integration efforts between Flink and Pulsar.</p>
+</description>
+<pubDate>Mon, 25 Nov 2019 13:00:00 +0100</pubDate>
+<link>https://flink.apache.org/news/2019/11/25/query-pulsar-streams-using-apache-flink.html</link>
+<guid isPermaLink="true">/news/2019/11/25/query-pulsar-streams-using-apache-flink.html</guid>
+</item>
+
+<item>
+<title>Apache Flink 1.9.1 Released</title>
+<description><p>The Apache Flink community released the first bugfix version of the Apache Flink 1.9 series.</p>
+
+<p>This release includes 96 fixes and minor improvements for Flink 1.9.0. The list below includes a detailed list of all fixes and improvements.</p>
+
+<p>We highly recommend all users to upgrade to Flink 1.9.1.</p>
+
+<p>Updated Maven dependencies:</p>
+
+<div class="highlight"><pre><code class="language-xml"><span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-java<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.9.1<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span>
+<span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-streaming-java_2.11<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.9.1<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span>
+<span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-clients_2.11<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.9.1<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span></code></pre></div>
+
+<p>You can find the binaries on the updated <a href="/downloads.html">Downloads page</a>.</p>
+
+<p>List of resolved issues:</p>
+
+<h2> Bug
+</h2>
+<ul>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11630">FLINK-11630</a>] - TaskExecutor does not wait for Task termination when terminating itself
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13490">FLINK-13490</a>] - Fix if one column value is null when reading JDBC, the following values are all null
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13941">FLINK-13941</a>] - Prevent data-loss by not cleaning up small part files from S3.
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12501">FLINK-12501</a>] - AvroTypeSerializer does not work with types generated by avrohugger
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13386">FLINK-13386</a>] - Fix some frictions in the new default Web UI
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13526">FLINK-13526</a>] - Switching to a non existing catalog or database crashes sql-client
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13568">FLINK-13568</a>] - DDL create table doesn&#39;t allow STRING data type
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13805">FLINK-13805</a>] - Bad Error Message when TaskManager is lost
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13806">FLINK-13806</a>] - Metric Fetcher floods the JM log with errors when TM is lost
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14010">FLINK-14010</a>] - Dispatcher &amp; JobManagers don&#39;t give up leadership when AM is shut down
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14145">FLINK-14145</a>] - CompletedCheckpointStore#getLatestCheckpoint(true) returns wrong checkpoint
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13059">FLINK-13059</a>] - Cassandra Connector leaks Semaphore on Exception and hangs on close
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13534">FLINK-13534</a>] - Unable to query Hive table with decimal column
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13562">FLINK-13562</a>] - Throws exception when FlinkRelMdColumnInterval meets two stage stream group aggregate
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13563">FLINK-13563</a>] - TumblingGroupWindow should implement toString method
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13564">FLINK-13564</a>] - Throw exception if constant with YEAR TO MONTH resolution was used for group windows
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13588">FLINK-13588</a>] - StreamTask.handleAsyncException throws away the exception cause
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13653">FLINK-13653</a>] - ResultStore should avoid using RowTypeInfo when creating a result
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13711">FLINK-13711</a>] - Hive array values not properly displayed in SQL CLI
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13737">FLINK-13737</a>] - flink-dist should add provided dependency on flink-examples-table
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13738">FLINK-13738</a>] - Fix NegativeArraySizeException in LongHybridHashTable
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13742">FLINK-13742</a>] - Fix code generation when aggregation contains both distinct aggregate with and without filter
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13760">FLINK-13760</a>] - Fix hardcode Scala version dependency in hive connector
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13761">FLINK-13761</a>] - `SplitStream` should be deprecated because `SplitJavaStream` is deprecated
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13789">FLINK-13789</a>] - Transactional Id Generation fails due to user code impacting formatting string
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13823">FLINK-13823</a>] - Incorrect debug log in CompileUtils
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13825">FLINK-13825</a>] - The original plugins dir is not restored after e2e test run
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13831">FLINK-13831</a>] - Free Slots / All Slots display error
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13887">FLINK-13887</a>] - Ensure defaultInputDependencyConstraint to be non-null when setting it in ExecutionConfig
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13897">FLINK-13897</a>] - OSS FS NOTICE file is placed in wrong directory
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13933">FLINK-13933</a>] - Hive Generic UDTF can not be used in table API both stream and batch mode
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13936">FLINK-13936</a>] - NOTICE-binary is outdated
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13966">FLINK-13966</a>] - Jar sorting in collect_license_files.sh is locale dependent
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14009">FLINK-14009</a>] - Cron jobs broken due to verifying incorrect NOTICE-binary file
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14049">FLINK-14049</a>] - Update error message for failed partition updates to include task name
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14076">FLINK-14076</a>] - &#39;ClassNotFoundException: KafkaException&#39; on Flink v1.9 w/ checkpointing
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14107">FLINK-14107</a>] - Kinesis consumer record emitter deadlock under event time alignment
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14119">FLINK-14119</a>] - Clean idle state for RetractableTopNFunction
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14139">FLINK-14139</a>] - Fix potential memory leak of rest server when using session/standalone cluster
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14140">FLINK-14140</a>] - The Flink Logo Displayed in Flink Python Shell is Broken
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14150">FLINK-14150</a>] - Unnecessary __pycache__ directories appears in pyflink.zip
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14288">FLINK-14288</a>] - Add Py4j NOTICE for source release
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13892">FLINK-13892</a>] - HistoryServerTest failed on Travis
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14043">FLINK-14043</a>] - SavepointMigrationTestBase is super slow
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12164">FLINK-12164</a>] - JobMasterTest.testJobFailureWhenTaskExecutorHeartbeatTimeout is unstable
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-9900">FLINK-9900</a>] - Fix unstable test ZooKeeperHighAvailabilityITCase#testRestoreBehaviourWithFaultyStateHandles
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13484">FLINK-13484</a>] - ConnectedComponents end-to-end test instable with NoResourceAvailableException
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13489">FLINK-13489</a>] - Heavy deployment end-to-end test fails on Travis with TM heartbeat timeout
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13514">FLINK-13514</a>] - StreamTaskTest.testAsyncCheckpointingConcurrentCloseAfterAcknowledge unstable
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13530">FLINK-13530</a>] - AbstractServerTest failed on Travis
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13585">FLINK-13585</a>] - Fix sporadical deallock in TaskAsyncCallTest#testSetsUserCodeClassLoader()
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13599">FLINK-13599</a>] - Kinesis end-to-end test failed on Travis
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13663">FLINK-13663</a>] - SQL Client end-to-end test for modern Kafka failed on Travis
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13688">FLINK-13688</a>] - HiveCatalogUseBlinkITCase.testBlinkUdf constantly failed
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13739">FLINK-13739</a>] - BinaryRowTest.testWriteString() fails in some environments
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13746">FLINK-13746</a>] - Elasticsearch (v2.3.5) sink end-to-end test fails on Travis
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13769">FLINK-13769</a>] - BatchFineGrainedRecoveryITCase.testProgram failed on Travis
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13807">FLINK-13807</a>] - Flink-avro unit tests fails if the character encoding in the environment is not default to UTF-8
+</li>
+</ul>
+
+<h2> Improvement
+</h2>
+<ul>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13965">FLINK-13965</a>] - Keep hasDeprecatedKeys and deprecatedKeys methods in ConfigOption and mark it with @Deprecated annotation
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-9941">FLINK-9941</a>] - Flush in ScalaCsvOutputFormat before close method
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13336">FLINK-13336</a>] - Remove the legacy batch fault tolerance page and redirect it to the new task failure recovery page
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13380">FLINK-13380</a>] - Improve the usability of Flink session cluster on Kubernetes
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13819">FLINK-13819</a>] - Introduce RpcEndpoint State
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13845">FLINK-13845</a>] - Drop all the content of removed &quot;Checkpointed&quot; interface
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13957">FLINK-13957</a>] - Log dynamic properties on job submission
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13967">FLINK-13967</a>] - Generate full binary licensing via collect_license_files.sh
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13968">FLINK-13968</a>] - Add travis check for the correctness of the binary licensing
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13449">FLINK-13449</a>] - Add ARM architecture to MemoryArchitecture
+</li>
+</ul>
+
+<h2> Documentation
+</h2>
+<ul>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13105">FLINK-13105</a>] - Add documentation for blink planner&#39;s built-in functions
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13277">FLINK-13277</a>] - add documentation of Hive source/sink
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13354">FLINK-13354</a>] - Add documentation for how to use blink planner
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13355">FLINK-13355</a>] - Add documentation for Temporal Table Join in blink planner
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13356">FLINK-13356</a>] - Add documentation for TopN and Deduplication in blink planner
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13359">FLINK-13359</a>] - Add documentation for DDL introduction
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13362">FLINK-13362</a>] - Add documentation for Kafka &amp; ES &amp; FileSystem DDL
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13363">FLINK-13363</a>] - Add documentation for streaming aggregate performance tunning.
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13706">FLINK-13706</a>] - add documentation of how to use Hive functions in Flink
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13942">FLINK-13942</a>] - Add Overview page for Getting Started section
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13863">FLINK-13863</a>] - Update Operations Playground to Flink 1.9.0
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13937">FLINK-13937</a>] - Fix wrong hive dependency version in documentation
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13830">FLINK-13830</a>] - The Document about Cluster on yarn have some problems
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-14160">FLINK-14160</a>] - Extend Operations Playground with --backpressure option
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13388">FLINK-13388</a>] - Update UI screenshots in the documentation to the new default Web Frontend
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13415">FLINK-13415</a>] - Document how to use hive connector in scala shell
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13517">FLINK-13517</a>] - Restructure Hive Catalog documentation
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13643">FLINK-13643</a>] - Document the workaround for users with a different minor Hive version
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13757">FLINK-13757</a>] - Fix wrong description of "IS NOT TRUE" function documentation
+</li>
+</ul>
+
+</description>
+<pubDate>Fri, 18 Oct 2019 14:00:00 +0200</pubDate>
+<link>https://flink.apache.org/news/2019/10/18/release-1.9.1.html</link>
+<guid isPermaLink="true">/news/2019/10/18/release-1.9.1.html</guid>
+</item>
+
+<item>
+<title>The State Processor API: How to Read, write and modify the state of Flink applications</title>
+<description><p>Whether you are running Apache Flink<sup>Ⓡ</sup> in production or evaluated Flink as a computation framework in the past, you’ve probably found yourself asking the question: How can I access, write or update state in a Flink savepoint? Ask no more! <a href="https://flink.apache.org/news/2019/08/22/release-1.9.0.html">Apache Flink 1.9.0</a> introduces the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.9/de [...]
+
+<p>In this post, we explain why this feature is a big step for Flink, what you can use it for, and how to use it. Finally, we will discuss the future of the State Processor API and how it aligns with our plans to evolve Flink into a system for <a href="https://flink.apache.org/news/2019/02/13/unified-batch-streaming-blink.html">unified batch and stream processing</a>.</p>
+
+<h2 id="stateful-stream-processing-with-apache-flink-until-flink-19">Stateful Stream Processing with Apache Flink until Flink 1.9</h2>
+
+<p>All non-trivial stream processing applications are stateful and most of them are designed to run for months or years. Over time, many of them accumulate a lot of valuable state that can be very expensive or even impossible to rebuild if it gets lost due to a failure. In order to guarantee the consistency and durability of application state, Flink featured a sophisticated checkpointing and recovery mechanism from very early on. With every release, the Flink community has added mo [...]
+
+<p>However, a feature that was commonly requested by Flink users was the ability to access the state of an application “from the outside”. This request was motivated by the need to validate or debug the state of an application, to migrate the state of an application to another application, to evolve an application from the Heap State Backend to the RocksDB State Backend, or to import the initial state of an application from an external system like a relational database.</p>
+
+<p>Despite all those convincing reasons to expose application state externally, your access options have been fairly limited until now. Flink’s Queryable State feature only supports key-lookups (point queries) and does not guarantee the consistency of returned values (the value of a key might be different before and after an application recovered from a failure). Moreover, queryable state cannot be used to add or modify the state of an application. Also, savepoints, which are consi [...]
+
+<h2 id="reading-and-writing-application-state-with-the-state-processor-api">Reading and Writing Application State with the State Processor API</h2>
+
+<p>The State Processor API that comes with Flink 1.9 is a true game-changer in how you can work with application state! In a nutshell, it extends the DataSet API with Input and OutputFormats to read and write savepoint or checkpoint data. Due to the <a href="https://ci.apache.org/projects/flink/flink-docs-master/dev/table/common.html#integration-with-datastream-and-dataset-api">interoperability of DataSet and Table API</a>, you can even use relational Table AP [...]
+
+<p>For example, you can take a savepoint of a running stream processing application and analyze it with a DataSet batch program to verify that the application behaves correctly. Or you can read a batch of data from any store, preprocess it, and write the result to a savepoint that you use to bootstrap the state of a streaming application. It’s also possible to fix inconsistent state entries now. Finally, the State Processor API opens up many ways to evolve a stateful application th [...]
+
+<h2 id="mapping-application-state-to-datasets">Mapping Application State to DataSets</h2>
+
+<p>The State Processor API maps the state of a streaming application to one or more data sets that can be separately processed. In order to be able to use the API, you need to understand how this mapping works.</p>
+
+<p>But let’s first have a look at what a stateful Flink job looks like. A Flink job is composed of operators, typically one or more source operators, a few operators for the actual processing, and one or more sink operators. Each operator runs in parallel in one or more tasks and can work with different types of state. An operator can have zero, one, or more <em>“operator states”</em> which are organized as lists that are scoped to the operator’s tasks. If the operator [...]
+
+<p>The following figure shows the application “MyApp” which consists of three operators called “Src”, “Proc”, and “Snk”. Src has one operator state (os1), Proc has one operator state (os2) and two keyed states (ks1, ks2) and Snk is stateless.</p>
+
+<p style="display: block; text-align: center; margin-top: 20px; margin-bottom: 20px">
+ <img src="/img/blog/2019-09-13-state-processor-api-blog/application-my-app-state-processor-api.png" width="600px" alt="Application: My App" />
+</p>
+
+<p>A savepoint or checkpoint of MyApp consists of the data of all states, organized in a way that the states of each task can be restored. When processing the data of a savepoint (or checkpoint) with a batch job, we need a mental model that maps the data of the individual tasks’ states into data sets or tables. In fact, we can think of a savepoint as a database. Every operator (identified by its UID) represents a namespace. Each operator state of an operator is mapped to a dedicate [...]
+
+<p style="display: block; text-align: center; margin-top: 20px; margin-bottom: 20px">
+ <img src="/img/blog/2019-09-13-state-processor-api-blog/database-my-app-state-processor-api.png" width="600px" alt="Database: My App" />
+</p>
+
+<p>The figure shows how the values of Src’s operator state are mapped to a table with one column and five rows, one row for all list entries across all parallel tasks of Src. Operator state os2 of the operator “Proc” is similarly mapped to an individual table. The keyed states ks1 and ks2 are combined to a single table with three columns, one for the key, one for ks1 and one for ks2. The keyed table holds one row for each distinct key of both keyed states. Since the operator “Snk” [...]
+
+<p>The State Processor API now offers methods to create, load, and write a savepoint. You can read a DataSet from a loaded savepoint or convert a DataSet into a state and add it to a savepoint. DataSets can be processed with the full feature set of the DataSet API. With these building blocks, all of the before-mentioned use cases (and more) can be addressed. Please have a look at the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/libs/state_process [...]
+
+<h2 id="why-dataset-api">Why DataSet API?</h2>
+
+<p>In case you are familiar with <a href="https://flink.apache.org/roadmap.html">Flink’s roadmap</a>, you might be surprised that the State Processor API is based on the DataSet API. The Flink community plans to extend the DataStream API with the concept of <em>BoundedStreams</em> and deprecate the DataSet API. When designing this feature, we also evaluated the DataStream API or Table API but neither could provide the right feature set yet. Since w [...]
+
+<h2 id="summary">Summary</h2>
+
+<p>Flink users have requested a feature to access and modify the state of streaming applications from the outside for a long time. With the State Processor API, Flink 1.9.0 finally exposes application state as a data format that can be manipulated. This feature opens up many new possibilities for how users can maintain and manage Flink streaming applications, including arbitrary evolution of stream applications and exporting and bootstrapping of application state. To put it concise [...]
+</description>
+<pubDate>Fri, 13 Sep 2019 14:00:00 +0200</pubDate>
+<link>https://flink.apache.org/feature/2019/09/13/state-processor-api.html</link>
+<guid isPermaLink="true">/feature/2019/09/13/state-processor-api.html</guid>
+</item>
+
+<item>
+<title>Apache Flink 1.8.2 Released</title>
+<description><p>The Apache Flink community released the second bugfix version of the Apache Flink 1.8 series.</p>
+
+<p>This release includes 23 fixes and minor improvements for Flink 1.8.1. The list below includes a detailed list of all fixes and improvements.</p>
+
+<p>We highly recommend all users to upgrade to Flink 1.8.2.</p>
+
+<p>Updated Maven dependencies:</p>
+
+<div class="highlight"><pre><code class="language-xml"><span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-java<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.8.2<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span>
+<span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-streaming-java_2.11<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.8.2<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span>
+<span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-clients_2.11<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.8.2<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span></code></pre></div>
+
+<p>You can find the binaries on the updated <a href="/downloads.html">Downloads page</a>.</p>
+
+<p>List of resolved issues:</p>
+
+<h2> Bug
+</h2>
+<ul>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13941">FLINK-13941</a>] - Prevent data-loss by not cleaning up small part files from S3.
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-9526">FLINK-9526</a>] - BucketingSink end-to-end test failed on Travis
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10368">FLINK-10368</a>] - &#39;Kerberized YARN on Docker test&#39; unstable
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12319">FLINK-12319</a>] - StackOverFlowError in cep.nfa.sharedbuffer.SharedBuffer
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12736">FLINK-12736</a>] - ResourceManager may release TM with allocated slots
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12889">FLINK-12889</a>] - Job keeps in FAILING state
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13059">FLINK-13059</a>] - Cassandra Connector leaks Semaphore on Exception; hangs on close
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13159">FLINK-13159</a>] - java.lang.ClassNotFoundException when restore job
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13367">FLINK-13367</a>] - Make ClosureCleaner detect writeReplace serialization override
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13369">FLINK-13369</a>] - Recursive closure cleaner ends up with stackOverflow in case of circular dependency
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13394">FLINK-13394</a>] - Use fallback unsafe secure MapR in nightly.sh
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13484">FLINK-13484</a>] - ConnectedComponents end-to-end test instable with NoResourceAvailableException
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13499">FLINK-13499</a>] - Remove dependency on MapR artifact repository
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13508">FLINK-13508</a>] - CommonTestUtils#waitUntilCondition() may attempt to sleep with negative time
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13586">FLINK-13586</a>] - Method ClosureCleaner.clean broke backward compatibility between 1.8.0 and 1.8.1
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13761">FLINK-13761</a>] - `SplitStream` should be deprecated because `SplitJavaStream` is deprecated
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13789">FLINK-13789</a>] - Transactional Id Generation fails due to user code impacting formatting string
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13806">FLINK-13806</a>] - Metric Fetcher floods the JM log with errors when TM is lost
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13807">FLINK-13807</a>] - Flink-avro unit tests fails if the character encoding in the environment is not default to UTF-8
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-13897">FLINK-13897</a>] - OSS FS NOTICE file is placed in wrong directory
+</li>
+</ul>
+
+<h2> Improvement
+</h2>
+<ul>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12578">FLINK-12578</a>] - Use secure URLs for Maven repositories
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12741">FLINK-12741</a>] - Update docs about Kafka producer fault tolerance guarantees
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12749">FLINK-12749</a>] - Add Flink Operations Playground documentation
+</li>
+</ul>
+</description>
+<pubDate>Wed, 11 Sep 2019 14:00:00 +0200</pubDate>
+<link>https://flink.apache.org/news/2019/09/11/release-1.8.2.html</link>
+<guid isPermaLink="true">/news/2019/09/11/release-1.8.2.html</guid>
+</item>
+
+<item>
+<title>Flink Community Update - September'19</title>
+<description><p>This has been an exciting, fast-paced year for the Apache Flink community. But with over 10k messages across the mailing lists, 3k Jira tickets and 2k pull requests, it is not easy to keep up with the latest state of the project. Plus everything happening around it. With that in mind, we want to bring back regular community updates to the Flink blog.</p>
+
+<p>The first post in the series takes you on an little detour across the year, to freshen up and make sure you’re all up to date.</p>
+
+<div class="page-toc">
+<ul id="markdown-toc">
+ <li><a href="#the-year-so-far-in-flink" id="markdown-toc-the-year-so-far-in-flink">The Year (so far) in Flink</a> <ul>
+ <li><a href="#integration-of-the-chinese-speaking-community" id="markdown-toc-integration-of-the-chinese-speaking-community">Integration of the Chinese-speaking community</a></li>
+ <li><a href="#improving-flinks-documentation" id="markdown-toc-improving-flinks-documentation">Improving Flink’s Documentation</a></li>
+ <li><a href="#adjusting-the-contribution-process-and-experience" id="markdown-toc-adjusting-the-contribution-process-and-experience">Adjusting the Contribution Process and Experience</a></li>
+ <li><a href="#new-committers-and-pmc-members" id="markdown-toc-new-committers-and-pmc-members">New Committers and PMC Members</a> <ul>
+ <li><a href="#new-pmc-members" id="markdown-toc-new-pmc-members">New PMC Members</a></li>
+ <li><a href="#new-committers" id="markdown-toc-new-committers">New Committers</a></li>
+ </ul>
+ </li>
+ </ul>
+ </li>
+ <li><a href="#the-bigger-picture" id="markdown-toc-the-bigger-picture">The Bigger Picture</a></li>
+ <li><a href="#upcoming-events" id="markdown-toc-upcoming-events">Upcoming Events</a> <ul>
+ <li><a href="#north-america" id="markdown-toc-north-america">North America</a></li>
+ <li><a href="#europe" id="markdown-toc-europe">Europe</a></li>
+ <li><a href="#asia" id="markdown-toc-asia">Asia</a></li>
+ </ul>
+ </li>
+</ul>
+
+</div>
+
+<h1 id="the-year-so-far-in-flink">The Year (so far) in Flink</h1>
+
+<p>Two major versions were released this year: <a href="https://flink.apache.org/news/2019/04/09/release-1.8.0.html">Flink 1.8</a> and <a href="https://flink.apache.org/news/2019/08/22/release-1.9.0.html">Flink 1.9</a>; paving the way for the goal of making Flink the first framework to seamlessly support stream and batch processing with a single, unified runtime. The <a href="https://flink.apache.org/news/2019/02/13/unified-batch- [...]
+
+<p>The 1.9 release was the result of the <strong>biggest community effort the project has experienced so far</strong>, with the number of contributors soaring to 190 (see <a href="#the-bigger-picture">The Bigger Picture</a>). For a quick overview of the upcoming work for Flink 1.10 (and beyond), have a look at the updated <a href="https://flink.apache.org/roadmap.html">roadmap</a>!</p>
+
+<h2 id="integration-of-the-chinese-speaking-community">Integration of the Chinese-speaking community</h2>
+
+<p>As the number of Chinese-speaking Flink users rapidly grows, the community is working on translating resources and creating dedicated spaces for discussion to invite and include these users in the wider Flink community. Part of the ongoing work is described in <a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-35%3A+Support+Chinese+Documents+and+Website">FLIP-35</a> and has resulted in:</p>
+
+<ul>
+ <li>A new user mailing list (user-zh@f.a.o) dedicated to Chinese-speakers.</li>
+</ul>
+<p></p>
+<ul>
+ <li>A Chinese translation of the Apache Flink <a href="https://flink.apache.org/zh/">website</a> and <a href="https://ci.apache.org/projects/flink/flink-docs-master/zh/">documentation</a>.</li>
+</ul>
+<p></p>
+<ul>
+ <li>Multiple meetups organized all over China, with the biggest one reaching a whopping number of 500+ participants. Some of these meetups were also organized in collaboration with communities from other projects, like Apache Pulsar and Apache Kafka.</li>
+</ul>
+
+<center>
+<img src="/img/blog/2019-09-05-flink-community-update/2019-09-05-flink-community-update_3.png" width="800px" alt="China Meetup" />
+</center>
+
+<p>In case you’re interested in knowing more about this work in progress, Robert Metzger and Fabian Hueske will be diving into “Inviting Apache Flink’s Chinese User Community” at the upcoming ApacheCon Europe 2019 (see <a href="#upcoming-flink-community-events">Upcoming Flink Community Events</a>).</p>
+
+<h2 id="improving-flinks-documentation">Improving Flink’s Documentation</h2>
+
+<p>Besides the translation effort, the community has also been working quite hard on a <strong>Flink docs overhaul</strong>. The main goals are to:</p>
+
+<ul>
+ <li>Organize and clean-up the structure of the docs;</li>
+</ul>
+<p></p>
+<ul>
+ <li>Align the content with the overall direction of the project;</li>
+</ul>
+<p></p>
+<ul>
+ <li>Improve the <em>getting-started</em> material and make the content more accessible to different levels of Flink experience.</li>
+</ul>
+
+<p>Given that there has been some confusion in the past regarding unclear definition of core Flink concepts, one of the first completed efforts was to introduce a <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.9/concepts/glossary.html#glossary">Glossary</a> in the docs. To get up to speed with the roadmap for the remainder efforts, you can refer to <a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-42%3A+Rework+Flink+Do [...]
+
+<h2 id="adjusting-the-contribution-process-and-experience">Adjusting the Contribution Process and Experience</h2>
+
+<p>The <a href="https://flink.apache.org/contributing/how-to-contribute.html">guidelines</a> to contribute to Apache Flink have been reworked on the website, in an effort to lower the entry barrier for new contributors and reduce the overall friction in the contribution process. In addition, the Flink community discussed and adopted <a href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=120731026">bylaws</a> to help the [...]
+
+<p>For code contributors, a <a href="https://flink.apache.org/contributing/code-style-and-quality-preamble.html">Code Style and Quality Guide</a> that captures the expected standards for contributions was also added to the “Contributing” section of the Flink website.</p>
+
+<p>It’s important to stress that <strong>contributions are not restricted to code</strong>. Non-code contributions such as mailing list support, documentation work or organization of community events are equally as important to the development of the project and highly encouraged.</p>
+
+<h2 id="new-committers-and-pmc-members">New Committers and PMC Members</h2>
+
+<p>The Apache Flink community has welcomed <strong>5 new Committers</strong> and <strong>4 PMC (Project Management Committee) Members</strong> in 2019, so far:</p>
+
+<h3 id="new-pmc-members">New PMC Members</h3>
+<div class="highlight"><pre><code>Jincheng Sun, Kete (Kurt) Young, Kostas Kloudas, Thomas Weise
+</code></pre></div>
+
+<h3 id="new-committers">New Committers</h3>
+<div class="highlight"><pre><code>Andrey Zagrebin, Hequn, Jiangjie (Becket) Qin, Rong Rong, Zhijiang Wang
+</code></pre></div>
+
+<p>Congratulations and thank you for your hardworking commitment to Flink!</p>
+
+<h1 id="the-bigger-picture">The Bigger Picture</h1>
+
+<p>Flink continues to push the boundaries of (stream) data processing, and the community is proud to see an ever-increasingly diverse set of contributors, users and technologies join the ecosystem.</p>
+
+<p>In the timeframe of three releases, the project jumped from <strong>112 to 190 contributors</strong>, also doubling down on the number of requested changes and improvements. To top it off, the Flink GitHub repository recently reached the milestone of <strong>10k stars</strong>, all the way up from the incubation days in 2014.</p>
+
+<center>
+<img src="/img/blog/2019-09-05-flink-community-update/2019-09-05-flink-community-update_1.png" width="1000px" alt="GitHub" />
+</center>
+
+<p>The activity across the user@ and dev@<sup>1</sup> mailing lists shows a healthy heartbeat, and the gradual ramp up of user-zh@ suggests that this was a well-received community effort. Looking at the numbers for the same period in 2018, the dev@ mailing list has seen the biggest surge in activity, with an average growth of <strong>2.5x in the number of messages and distinct users</strong> — a great reflection of the hyperactive pace of development of the [...]
+
+<p><img style="float: right;" src="/img/blog/2019-09-05-flink-community-update/2019-09-05-flink-community-update_2.png" width="420px" alt="Mailing Lists" /></p>
+
+<p>In support of these observations, the report for the financial year of 2019 from the Apache Software Foundation (ASF) features Flink as one of the most thriving open source projects, with mentions for:</p>
+
+<ul>
+ <li>Most Active Visits and Downloads</li>
+</ul>
+<p></p>
+<ul>
+ <li>Most Active Sources: Visits</li>
+</ul>
+<p></p>
+<ul>
+ <li>Most Active Sources: Clones</li>
+</ul>
+<p></p>
+<ul>
+ <li>Top Repositories by Number of Commits</li>
+</ul>
+<p></p>
+<ul>
+ <li>Top Most Active Apache Mailing Lists (user@ and dev@)</li>
+</ul>
+
+<p>Hats off to our fellows at Apache Beam for an astounding year, too! For more detailed insights, check the <a href="https://s3.amazonaws.com/files-dist/AnnualReports/FY2018%20Annual%20Report.pdf">full report</a>.</p>
+<p></p>
+<p><sup>1. Excluding messages from “jira@apache.org”.</sup></p>
+
+<h1 id="upcoming-events">Upcoming Events</h1>
+
+<p>As the conference and meetup season ramps up again, here are some events to keep an eye out for talks about Flink and opportunities to mingle with the wider stream processing community.</p>
+
+<h3 id="north-america">North America</h3>
+
+<ul>
+ <li>[Conference] <strong><a href="https://conferences.oreilly.com/strata/strata-ny">Strata Data Conference 2019</a></strong>, September 23-26, New York, USA
+ <p></p>
+ </li>
+ <li>[Meetup] <strong><a href="https://www.meetup.com/Bay-Area-Apache-Flink-Meetup/events/262680261/">Apache Flink Bay Area Meetup</a></strong>, September 24, San Francisco, USA
+ <p></p>
+ </li>
+ <li>[Conference] <strong><a href="https://www.meetup.com/Bay-Area-Apache-Flink-Meetup/events/262680261/">Scale By The Bay 2019</a></strong>, November 13-15, San Francisco, USA</li>
+</ul>
+
+<h3 id="europe">Europe</h3>
+
+<ul>
+ <li>[Meetup] <strong><a href="https://www.meetup.com/Apache-Flink-London-Meetup/events/264123672">Apache Flink London Meetup</a></strong>, September 23, London, UK
+ <p></p>
+ </li>
+ <li>[Conference] <strong><a href="https://europe-2019.flink-forward.org">Flink Forward Europe 2019</a></strong>, October 7-9, Berlin, Germany
+ <p></p>
+ <ul>
+ <li>
+ <p>The next edition of Flink Forward Europe is around the corner and the <a href="https://europe-2019.flink-forward.org/conference-program">program</a> has been announced, featuring 70+ talks as well as panel discussions and interactive “Ask Me Anything” sessions with core Flink committers. If you’re looking to learn more about Flink and share your experience with other community members, there really is <a href="(https://vimeo.com/296403091)&q [...]
+ </li>
+ <li>
+ <p><strong>Note:</strong> if you are a <strong>committer for any Apache project</strong>, you can <strong>get a free ticket</strong> by registering with your Apache email address and using the discount code: <em>FFEU19-ApacheCommitter</em>.</p>
+ </li>
+ </ul>
+ </li>
+</ul>
+<p></p>
+<ul>
+ <li>[Conference] <strong><a href="https://aceu19.apachecon.com/">ApacheCon Berlin 2019</a></strong>, October 22-24, Berlin, Germany</li>
+</ul>
+<p></p>
+<ul>
+ <li>[Conference] <strong><a href="https://www.data2day.de/">Data2Day 2019</a></strong>, October 22-24, Ludwigshafen, Germany</li>
+</ul>
+<p></p>
+<ul>
+ <li>[Conference] <strong><a href="https://bigdatatechwarsaw.eu">Big Data Tech Warsaw 2020</a></strong>, February 7, Warsaw, Poland
+ <p></p>
+ <ul>
+ <li>The Call For Presentations (CFP) is now <a href="https://bigdatatechwarsaw.eu/cfp/">open</a>.</li>
+ </ul>
+ </li>
+</ul>
+
+<h3 id="asia">Asia</h3>
+
+<ul>
+ <li>[Conference] <strong><a href="https://m.aliyun.com/markets/aliyun/developer/ffa2019">Flink Forward Asia 2019</a></strong>, November 28-30, Beijing, China
+ <p></p>
+ <ul>
+ <li>The second edition of Flink Forward Asia is also happening later this year, in Beijing, and the CFP is <a href="https://developer.aliyun.com/special/ffa2019">open</a> until September 20.</li>
+ </ul>
+ </li>
+</ul>
+
+<p>If you’d like to keep a closer eye on what’s happening in the community, subscribe to the <a href="https://flink.apache.org/community.html#mailing-lists">community mailing list</a> to get fine-grained weekly updates, upcoming event announcements and more. Also, please reach out if you’re interested in organizing or being part of Flink events in your area!</p>
+</description>
+<pubDate>Tue, 10 Sep 2019 14:00:00 +0200</pubDate>
+<link>https://flink.apache.org/news/2019/09/10/community-update.html</link>
+<guid isPermaLink="true">/news/2019/09/10/community-update.html</guid>
+</item>
+
+<item>
+<title>Apache Flink 1.9.0 Release Announcement</title>
+<description><p>The Apache Flink community is proud to announce the release of Apache Flink
+1.9.0.</p>
+
+<p>The Apache Flink project’s goal is to develop a stream processing system to
+unify and power many forms of real-time and offline data processing
+applications as well as event-driven applications. In this release, we have
+made a huge step forward in that effort, by integrating Flink’s stream and
+batch processing capabilities under a single, unified runtime.</p>
+
+<p>Significant features on this path are batch-style recovery for batch jobs and
+a preview of the new Blink-based query engine for Table API and SQL queries.
+We are also excited to announce the availability of the State Processor API,
+which is one of the most frequently requested features and enables users to
+read and write savepoints with Flink DataSet jobs. Finally, Flink 1.9 includes
+a reworked WebUI and previews of Flink’s new Python Table API and its
+integration with the Apache Hive ecosystem.</p>
+
+<p>This blog post describes all major new features and improvements, important
+changes to be aware of and what to expect moving forward. For more details,
+check the <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&amp;version=12344601">complete release
+changelog</a>.</p>
+
+<p>The binary distribution and source artifacts for this release are now
+available via the <a href="https://flink.apache.org/downloads.html">Downloads</a> page of
+the Flink project, along with the updated
+<a href="https://ci.apache.org/projects/flink/flink-docs-release-1.9/">documentation</a>.
+Flink 1.9 is API-compatible with previous 1.x releases for APIs annotated with
+the <code>@Public</code> annotation.</p>
+
+<p>Please feel encouraged to download the release and share your thoughts with
+the community through the Flink <a href="https://flink.apache.org/community.html#mailing-lists">mailing
+lists</a> or
+<a href="https://issues.apache.org/jira/projects/FLINK/summary">JIRA</a>. As always,
+feedback is very much appreciated!</p>
+
+<div class="page-toc">
+<ul id="markdown-toc">
+ <li><a href="#new-features-and-improvements" id="markdown-toc-new-features-and-improvements">New Features and Improvements</a> <ul>
+ <li><a href="#fine-grained-batch-recovery-flip-1" id="markdown-toc-fine-grained-batch-recovery-flip-1">Fine-grained Batch Recovery (FLIP-1)</a></li>
+ <li><a href="#state-processor-api-flip-43" id="markdown-toc-state-processor-api-flip-43">State Processor API (FLIP-43)</a></li>
+ <li><a href="#stop-with-savepoint-flip-34" id="markdown-toc-stop-with-savepoint-flip-34">Stop-with-Savepoint (FLIP-34)</a></li>
+ <li><a href="#flink-webui-rework" id="markdown-toc-flink-webui-rework">Flink WebUI Rework</a></li>
+ <li><a href="#preview-of-the-new-blink-sql-query-processor" id="markdown-toc-preview-of-the-new-blink-sql-query-processor">Preview of the new Blink SQL Query Processor</a></li>
+ <li><a href="#preview-of-full-hive-integration-flink-10556" id="markdown-toc-preview-of-full-hive-integration-flink-10556">Preview of Full Hive Integration (FLINK-10556)</a></li>
+ <li><a href="#preview-of-the-new-python-table-api-flip-38" id="markdown-toc-preview-of-the-new-python-table-api-flip-38">Preview of the new Python Table API (FLIP-38)</a></li>
+ </ul>
+ </li>
+ <li><a href="#important-changes" id="markdown-toc-important-changes">Important Changes</a></li>
+ <li><a href="#release-notes" id="markdown-toc-release-notes">Release Notes</a></li>
+ <li><a href="#list-of-contributors" id="markdown-toc-list-of-contributors">List of Contributors</a></li>
+</ul>
+
+</div>
+
+<h2 id="new-features-and-improvements">New Features and Improvements</h2>
+
+<h3 id="fine-grained-batch-recovery-flip-1">Fine-grained Batch Recovery (FLIP-1)</h3>
+
+<p>The time to recover a batch (DataSet, Table API and SQL) job from a task
+failure was significantly reduced. Until Flink 1.9, task failures in batch
+jobs were recovered by canceling all tasks and restarting the whole job, i.e,
+the job was started from scratch and all progress was voided. With this
+release, Flink can be configured to limit the recovery to only those tasks
+that are in the same <strong>failover region</strong>. A failover region is the set of
+tasks that are connected via pipelined data exchanges. Hence, the
+batch-shuffle connections of a job define the boundaries of its failover
+regions. More details are available in
+<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-1+%3A+Fine+Grained+Recovery+from+Task+Failures">FLIP-1</a>.
+<img src="/img/blog/release-19-flip1.png" alt="alt_text" title="Fine-grained Batch
+Recovery" /></p>
+
+<p>To use this new failover strategy, you need to do the following
+settings:</p>
+
+<ul>
+ <li>Make sure you have the entry <code>jobmanager.execution.failover-strategy:
+region</code> in your <code>flink-conf.yaml</code>.</li>
+</ul>
+
+<p><strong>Note:</strong> The configuration of the 1.9 distribution has that entry by default,
+ but when reusing a configuration file from previous setups, you have to add
+ it manually.</p>
+
+<p>Moreover, you need to set the <code>ExecutionMode</code> of batch jobs in the
+<code>ExecutionConfig</code> to <code>BATCH</code> to configure that data shuffles are not pipelined
+and jobs have more than one failover region.</p>
+
+<p>The “Region” failover strategy also improves the recovery of “embarrassingly
+parallel” streaming jobs, i.e., jobs without any shuffle like keyBy() or
+rebalance. When such a job is recovered, only the tasks of the affected
+pipeline (failover region) are restarted. For all other streaming jobs, the
+recovery behavior is the same as in prior Flink versions.</p>
+
+<h3 id="state-processor-api-flip-43">State Processor API (FLIP-43)</h3>
+
+<p>Up to Flink 1.9, accessing the state of a job from the outside was limited to
+the (still) experimental <a href="https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/queryable_state.html">Queryable
+State</a>.
+This release introduces a new, powerful library to read, write and modify
+state snapshots using the batch DataSet API. In practice, this means:</p>
+
+<ul>
+ <li>Flink job state can be bootstrapped by reading data from external systems,
+such as external databases, and converting it into a savepoint.</li>
+ <li>State in savepoints can be queried using any of Flink’s batch APIs
+(DataSet, Table, SQL), for example to analyze relevant state patterns or
+check for discrepancies in state that can support application auditing or
+troubleshooting.</li>
+ <li>The schema of state in savepoints can be migrated offline, compared to the
+previous approach requiring online migration on schema access.</li>
+ <li>Invalid data in savepoints can be identified and corrected.</li>
+</ul>
+
+<p>The new State Processor API covers all variations of snapshots: savepoints,
+full checkpoints and incremental checkpoints. More details are available in
+<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-43%3A+State+Processor+API">FLIP-43</a></p>
+
+<h3 id="stop-with-savepoint-flip-34">Stop-with-Savepoint (FLIP-34)</h3>
+
+<p><a href="https://ci.apache.org/projects/flink/flink-docs-stable/ops/state/savepoints.html#operations">Cancelling with a
+savepoint</a>
+is a common operation for stopping/restarting, forking or updating Flink jobs.
+However, the existing implementation did not guarantee output persistence to
+external storage systems for exactly-once sinks. To improve the end-to-end
+semantics when stopping a job, Flink 1.9 introduces a new <code>SUSPEND</code> mode to
+stop a job with a savepoint that is consistent with the emitted data.
+You can suspend a job with Flink’s CLI client as follows:</p>
+
+<div class="highlight"><pre><code>bin/flink stop -p [:targetDirectory] :jobId
+</code></pre></div>
+
+<p>The final job state is set to <code>FINISHED</code> on success, allowing
+users to detect failures of the requested operation.</p>
+
+<p>More details are available in
+<a href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=103090212">FLIP-34</a></p>
+
+<h3 id="flink-webui-rework">Flink WebUI Rework</h3>
+
+<p>After a
+<a href="http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Change-underlying-Frontend-Architecture-for-Flink-Web-Dashboard-td24902.html">discussion</a>
+about modernizing the internals of Flink’s WebUI, this component was
+reconstructed using the latest stable version of Angular — basically, a bump
+from Angular 1.x to 7.x. The redesigned version is the default in 1.9.0,
+however there is a link to switch to the old WebUI.</p>
+
+<div class="row"> <div class="col-sm-6"> <span><img class="thumbnail" src="/img/blog/release-19-web1.png" /></span> </div> <div class="col-sm-6"> <span><img class="thumbnail" src="/img/blog/release-19-web2.png" /></span> </div>
+ </div>
+
+<p><strong>Note:</strong> Moving forward, feature parity for the old version of the WebUI
+will not be guaranteed.</p>
+
+<h3 id="preview-of-the-new-blink-sql-query-processor">Preview of the new Blink SQL Query Processor</h3>
+
+<p>Following the <a href="/news/2019/02/13/unified-batch-streaming-blink.html">donation of
+Blink</a> to
+Apache Flink, the community worked on integrating Blink’s query optimizer and
+runtime for the Table API and SQL. As a first step, we refactored the
+monolithic <code>flink-table</code> module into smaller modules
+(<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions">FLIP-32</a>).
+This resulted in a clear separation of and well-defined interfaces between the
+Java and Scala API modules and the optimizer and runtime modules.</p>
+
+<p><span><img style="width:50%" src="/img/blog/release-19-stack.png" /></span></p>
+
+<p>Next, we extended Blink’s planner to implement the new optimizer interface
+such that there are now two pluggable query processors to execute Table API
+and SQL statements: the pre-1.9 Flink processor and the new Blink-based query
+processor. The Blink-based query processor offers better SQL coverage (full TPC-H
+coverage in 1.9, TPC-DS coverage is planned for the next release) and improved
+performance for batch queries as the result of more extensive query
+optimization (cost-based plan selection and more optimization rules), improved
+code-generation, and tuned operator implementations.
+The Blink-based query processor also provides a more powerful streaming runner,
+with some new features (e.g. dimension table join, TopN, deduplication) and
+optimizations to solve data-skew in aggregation and more useful built-in
+functions.</p>
+
+<p><strong>Note:</strong> The semantics and set of supported operations of the query
+processors are mostly, but not fully aligned.</p>
+
+<p>However, the integration of Blink’s query processor is not fully completed
+yet. Therefore, the pre-1.9 Flink processor is still the default processor in
+Flink 1.9 and recommended for production settings. You can enable the Blink
+processor by configuring it via the <code>EnvironmentSettings</code> when creating a
+<code>TableEnvironment</code>. The selected processor must be on the classpath of the
+executing Java process. For cluster setups, both query processors are
+automatically loaded with the default configuration. When running a query from
+your IDE you need to explicitly <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/table/#table-program-dependencies">add a planner
+dependency</a>
+to your project.</p>
+
+<h4 id="other-improvements-to-the-table-api-and-sql"><strong>Other Improvements to the Table API and SQL</strong></h4>
+
+<p>Besides the exciting progress around the Blink planner, the community worked
+on a whole set of other improvements to these interfaces, including:</p>
+
+<ul>
+ <li>
+ <p><strong>Scala-free Table API and SQL for Java users
+(<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions">FLIP-32</a>)</strong></p>
+
+ <p>As part of the refactoring and splitting of the flink-table module, two
+separate API modules for Java and Scala were created. For Scala users,
+nothing really changes, but Java users can use the Table API and/or SQL now
+without pulling in a Scala dependency.</p>
+ </li>
+ <li>
+ <p><strong>Rework of the Table API Type System</strong>
+<strong>(<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-37%3A+Rework+of+the+Table+API+Type+System">FLIP-37</a>)</strong></p>
+
+ <p>The community implemented a <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/table/types.html#data-types">new data type
+system</a>
+to detach the Table API from Flink’s
+<a href="https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/types_serialization.html#flinks-typeinformation-class">TypeInformation</a>
+class and improve its compliance with the SQL standard. This is still a
+work in progress and expected to be completed in the next release. In
+Flink 1.9, UDFs are―among other things―not ported to the new type system
+yet.</p>
+ </li>
+ <li>
+ <p><strong>Multi-column and Multi-row Transformations for Table API</strong>
+<strong>(<a href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=97552739">FLIP-29</a>)</strong></p>
+
+ <p>The functionality of the Table API was extended with a set of
+transformations that support multi-row and/or multi-column inputs and
+outputs. These transformations significantly ease the implementation of
+processing logic that would be cumbersome to implement with relational
+operators.</p>
+ </li>
+ <li>
+ <p><strong>New, Unified Catalog APIs</strong>
+<strong>(<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-30%3A+Unified+Catalog+APIs">FLIP-30</a>)</strong></p>
+
+ <p>We reworked the catalog APIs to store metadata and unified the handling of
+internal and external catalogs. This effort was mainly initiated as a
+prerequisite for the Hive integration (see below), but improves the overall
+convenience of managing catalog metadata in Flink. Besides improving the
+catalog interfaces, we also extended their functionality. Previously table
+definitions for Table API or SQL queries were volatile. With Flink 1.9, the
+metadata of tables which are registered with a SQL DDL statement can be
+persisted in a catalog. This means you can add a table that is backed by a
+Kafka topic to a Metastore catalog and from then on query this table
+whenever your catalog is connected to Metastore.</p>
+ </li>
+ <li>
+ <p><strong>DDL Support in the SQL API
+(<a href="https://issues.apache.org/jira/browse/FLINK-10232">FLINK-10232</a>)</strong></p>
+
+ <p>Up to this point, Flink SQL only supported DML statements (e.g. <code>SELECT</code>,
+<code>INSERT</code>). External tables (table sources and sinks) had to be registered
+via Java/Scala code or configuration files. For 1.9, we added support for
+SQL DDL statements to register and remove tables and views (<code>CREATE TABLE,
+DROP TABLE)</code>. However, we did not add
+stream-specific syntax extensions to define timestamp extraction and
+watermark generation, yet. Full support for streaming use cases is planned
+for the next release.</p>
+ </li>
+</ul>
+
+<h3 id="preview-of-full-hive-integration-flink-10556">Preview of Full Hive Integration (FLINK-10556)</h3>
+
+<p>Apache Hive is widely used in Hadoop’s ecosystem to store and query large
+amounts of structured data. Besides being a query processor, Hive features a
+catalog called Metastore to manage and organize large datasets. A common
+integration point for query processors is to integrate with Hive’s Metastore
+in order to be able to tap into the data managed by Hive.</p>
+
+<p>Recently, the community started implementing an external catalog for Flink’s
+Table API and SQL that connects to Hive’s Metastore. In Flink 1.9, users will
+be able to query and process all data that is stored in Hive. As described
+earlier, you will also be able to persist metadata of Flink tables in Metastore.
+Moreover, the Hive integration includes support to use Hive’s UDFs in Flink
+Table API or SQL queries. More details are available in
+<a href="https://issues.apache.org/jira/browse/FLINK-10556">FLINK-10556</a>.</p>
+
+<p>While, previously, table definitions for Table API or SQL queries were always
+volatile, the new catalog connector additionally allows persisting a table in
+Metastore that is created with a SQL DDL statement (see above). This means
+that you connect to Metastore and register a table that is, for example,
+backed by a Kafka topic. From now on, you can query that table whenever your
+catalog is connected to Metastore.</p>
+
+<p>Please note that the Hive support in Flink 1.9 is experimental. We are
+planning to stabilize these features for the next release and are looking
+forward to your feedback.</p>
+
+<h3 id="preview-of-the-new-python-table-api-flip-38">Preview of the new Python Table API (FLIP-38)</h3>
+
+<p>This release also introduces a first version of a Python Table API
+(<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-38%3A+Python+Table+API">FLIP-38</a>).
+This marks the start towards our goal of bringing
+full-fledged Python support to Flink. The feature was designed as a slim
+Python API wrapper around the Table API, basically translating Python Table
+API method calls into Java Table API calls. In the initial version that ships
+with Flink 1.9, the Python Table API does not support UDFs yet, but just
+standard relational operations. Support for UDFs implemented in Python is on
+the roadmap for future releases.</p>
+
+<p>If you’d like to try the new Python API, you have to manually <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.9/flinkDev/building.html#build-pyflink">install
+PyFlink</a>.
+From there, you can have a look at <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.9/tutorials/python_table_api.html">this
+walkthrough</a>
+or explore it on your own. The <a href="http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-Publish-the-PyFlink-into-PyPI-td31201.html">community is currently
+working</a>
+on preparing a <code>pyflink</code> Python package that will be made available for
+installation via <code>pip</code>.</p>
+
+<h2 id="important-changes">Important Changes</h2>
+
+<ul>
+ <li>The Table API and SQL are now part of the default configuration of the
+Flink distribution. Before, the Table API and SQL had to be enabled by
+moving the corresponding JAR file from ./opt to ./lib.</li>
+ <li>The machine learning library (flink-ml) has been removed in preparation for
+<a href="https://docs.google.com/document/d/1StObo1DLp8iiy0rbukx8kwAJb0BwDZrQrMWub3DzsEo/edit">FLIP-39</a>.</li>
+ <li>The old DataSet and DataStream Python APIs have been removed in favor of
+<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-38%3A+Python+Table+API">FLIP-38</a>.</li>
+ <li>Flink can be compiled and run on Java 9. Note that certain components
+interacting with external systems (connectors, filesystems, reporters) may
+not work since the respective projects may have skipped Java 9 support.</li>
+</ul>
+
+<h2 id="release-notes">Release Notes</h2>
+
+<p>Please review the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.9/release-notes/flink-1.9.html">release
+notes</a>
+for a more detailed list of changes and new features if you plan to upgrade
+your Flink setup to Flink 1.9.0.</p>
+
+<h2 id="list-of-contributors">List of Contributors</h2>
+
+<p>We would like to thank all contributors who have made this release possible:</p>
+
+<p>Abdul Qadeer (abqadeer), Aitozi, Alberto Romero, Aleksey Pak, Alexander
+Fedulov, Alice Yan, Aljoscha Krettek, Aloys, Andrew Duffy, Andrey Zagrebin,
+Ankur, Artsem Semianenka, Benchao Li, Biao Liu, Bo WANG, Bowen L, Chesnay
+Schepler, Clark Yang, Congxian Qiu, Cristian, Danny Chan, David Moravek, Dawid
+Wysakowicz, Dian Fu, EronWright, Fabian Hueske, Fabio Lombardelli, Fokko
+Driesprong, Gao Yun, Gary Yao, Gen Luo, Gyula Fora, Hequn Cheng,
+Hongtao Zhang, Huang Xingbo, HuangXingBo, Hugo Da Cruz Louro, Humberto
+Rodríguez A, Hwanju Kim, Igal Shilman, Jamie Grier, Jark Wu, Jason, Jasper
+Yue, Jeff Zhang, Jiangjie (Becket) Qin, Jiezhi.G, Jincheng Sun, Jing Zhang,
+Jingsong Lee, Juan Gentile, Jungtaek Lim, Kailash Dayanand, Kevin
+Bohinski, Konstantin Knauf, Konstantinos Papadopoulos, Kostas Kloudas, Kurt
+Young, Lakshmi, Lakshmi Gururaja Rao, Leeviiii, LouisXu, Maximilian Michels,
+Nico Kruber, Niels Basjes, Paul Lam, PengFei Li, Peter Huang, Pierre Zemb,
+Piotr Nowojski, Piyush Narang, Richard Deurwaarder, Robert Metzger, Robert
+Stoll, Romano Vacca, Rong Rong, Rui Li, Ryantaocer, Scott Mitchell, Seth
+Wiesman, Shannon Carey, Shimin Yang, Stefan Richter, Stephan Ewen, Stephen
+Connolly, Steven Wu, SuXingLee, TANG Wen-hui, Thomas Weise, Till Rohrmann,
+Timo Walther, Tom Goong, TsReaper, Tzu-Li (Gordon) Tai, Ufuk Celebi,
+Victor Wong, WangHengwei, Wei Zhong, WeiZhong94, Xintong Song, Xpray,
+XuQianJin-Stars, Xuefu Zhang, Xupingyong, Yangze Guo, Yu Li, Yun Gao, Yun
+Tang, Zhanchun Zhang, Zhenghua Gao, Zhijiang, Zhu Zhu, Zili
+Chen, aloys, arganzheng, azagrebin, bd2019us, beyond1920, biao.liub,
+blueszheng, boshu Zheng, chenqi, chummyhe89, chunpinghe, dcadmin,
+dianfu, godfrey he, guanghui01.rong, hehuiyuan, hello, hequn8128,
+jackyyin, joongkeun.yang, klion26, lamber-ken, leesf, liguowei,
+lincoln-lil, liyafan82, luoqi, mans2singh, maqingxiang, maxin, mjl, okidogi,
+ozan, potseluev, qiangsi.lq, qiaoran, robbinli, shaoxuan-wang, shengqian.zhou,
+shenlang.sl, shuai-xu, sunhaibotb, tianchen, tianchen92,
+tison, tom_gong, vinoyang, vthinkxie, wanggeng3, wenhuitang, winifredtamg,
+xl38154, xuyang1706, yangfei5, yanghua, yuzhao.cyz,
+zhangxin516, zhangxinxing, zhaofaxian, zhijiang, zjuwangg, 林小铂,
+黄培松, 时无两丶.</p>
+</description>
+<pubDate>Thu, 22 Aug 2019 04:30:00 +0200</pubDate>
+<link>https://flink.apache.org/news/2019/08/22/release-1.9.0.html</link>
+<guid isPermaLink="true">/news/2019/08/22/release-1.9.0.html</guid>
+</item>
+
+<item>
+<title>Flink Network Stack Vol. 2: Monitoring, Metrics, and that Backpressure Thing</title>
+<description><style type="text/css">
+.tg {border-collapse:collapse;border-spacing:0;}
+.tg td{padding:10px 10px;border-style:solid;border-width:1px;overflow:hidden;word-break:normal;}
+.tg th{padding:10px 10px;border-style:solid;border-width:1px;overflow:hidden;word-break:normal;background-color:#eff0f1;}
+.tg .tg-wide{padding:10px 30px;}
+.tg .tg-top{vertical-align:top}
+.tg .tg-topcenter{text-align:center;vertical-align:top}
+.tg .tg-center{text-align:center;vertical-align:center}
+</style>
+
+<p>In a <a href="/2019/06/05/flink-network-stack.html">previous blog post</a>, we presented how Flink’s network stack works from the high-level abstractions to the low-level details. This second blog post in the series of network stack posts extends on this knowledge and discusses monitoring network-related metrics to identify effects such as backpressure or bottlenecks in throughput and latency. Although this post briefly covers what to do with backpressure, [...]
+
+<div class="page-toc">
+<ul id="markdown-toc">
+ <li><a href="#monitoring" id="markdown-toc-monitoring">Monitoring</a> <ul>
+ <li><a href="#backpressure-monitor" id="markdown-toc-backpressure-monitor">Backpressure Monitor</a></li>
+ </ul>
+ </li>
+ <li><a href="#network-metrics" id="markdown-toc-network-metrics">Network Metrics</a> <ul>
+ <li><a href="#backpressure" id="markdown-toc-backpressure">Backpressure</a></li>
+ <li><a href="#resource-usage--throughput" id="markdown-toc-resource-usage--throughput">Resource Usage / Throughput</a></li>
+ <li><a href="#latency-tracking" id="markdown-toc-latency-tracking">Latency Tracking</a></li>
+ </ul>
+ </li>
+ <li><a href="#conclusion" id="markdown-toc-conclusion">Conclusion</a></li>
+</ul>
+
+</div>
+
+<h2 id="monitoring">Monitoring</h2>
+
+<p>Probably the most important part of network monitoring is <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/monitoring/back_pressure.html">monitoring backpressure</a>, a situation where a system is receiving data at a higher rate than it can process¹. Such behaviour will result in the sender being backpressured and may be caused by two things:</p>
+
+<ul>
+ <li>
+ <p>The receiver is slow.<br />
+This can happen because the receiver is backpressured itself, is unable to keep processing at the same rate as the sender, or is temporarily blocked by garbage collection, lack of system resources, or I/O.</p>
+ </li>
+ <li>
+ <p>The network channel is slow.<br />
+ Even though in such case the receiver is not (directly) involved, we call the sender backpressured due to a potential oversubscription on network bandwidth shared by all subtasks running on the same machine. Beware that, in addition to Flink’s network stack, there may be more network users, such as sources and sinks, distributed file systems (checkpointing, network-attached storage), logging, and metrics. A previous <a href="https://www.ververica.com/blog/how-to-size-your-apach [...]
+ </li>
+</ul>
+
+<p><sup>1</sup> In case you are unfamiliar with backpressure and how it interacts with Flink, we recommend reading through <a href="https://www.ververica.com/blog/how-flink-handles-backpressure">this blog post on backpressure</a> from 2015.</p>
+
+<p><br />
+If backpressure occurs, it will bubble upstream and eventually reach your sources and slow them down. This is not a bad thing per-se and merely states that you lack resources for the current load. However, you may want to improve your job so that it can cope with higher loads without using more resources. In order to do so, you need to find (1) where (at which task/operator) the bottleneck is and (2) what is causing it. Flink offers two mechanisms for identifying where the bottleneck is: [...]
+
+<ul>
+ <li>directly via Flink’s web UI and its backpressure monitor, or</li>
+ <li>indirectly through some of the network metrics.</li>
+</ul>
+
+<p>Flink’s web UI is likely the first entry point for a quick troubleshooting but has some disadvantages that we will explain below. On the other hand, Flink’s network metrics are better suited for continuous monitoring and reasoning about the exact nature of the bottleneck causing backpressure. We will cover both in the sections below. In both cases, you need to identify the origin of backpressure from the sources to the sinks. Your starting point for the current and future invest [...]
+
+<h3 id="backpressure-monitor">Backpressure Monitor</h3>
+
+<p>The <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/monitoring/back_pressure.html">backpressure monitor</a> is only exposed via Flink’s web UI². Since it’s an active component that is only triggered on request, it is currently not available via metrics. The backpressure monitor samples the running tasks’ threads on all TaskManagers via <code>Thread.getStackTrace()</code> and computes the number of samples where tasks were bl [...]
+
+<ul>
+ <li><span style="color:green">OK</span> for <code>ratio ≤ 0.10</code>,</li>
+ <li><span style="color:orange">LOW</span> for <code>0.10 &lt; Ratio ≤ 0.5</code>, and</li>
+ <li><span style="color:red">HIGH</span> for <code>0.5 &lt; Ratio ≤ 1</code>.</li>
+</ul>
+
+<p>Although you can tune things like the refresh-interval, the number of samples, or the delay between samples, normally, you would not need to touch these since the defaults already give good-enough results.</p>
+
+<center>
+<img src="/img/blog/2019-07-23-network-stack-2/back_pressure_sampling_high.png" width="600px" alt="Backpressure sampling:high" />
+</center>
+
+<p><sup>2</sup> You may also access the backpressure monitor via the REST API: <code>/jobs/:jobid/vertices/:vertexid/backpressure</code></p>
+
+<p><br />
+The backpressure monitor can help you find where (at which task/operator) backpressure originates from. However, it does not support you in further reasoning about the causes of it. Additionally, for larger jobs or higher parallelism, the backpressure monitor becomes too crowded to use and may also take some time to gather all information from all TaskManagers. Please also note that sampling may affect your running job’s performance.</p>
+
+<h2 id="network-metrics">Network Metrics</h2>
+
+<p><a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/monitoring/metrics.html#network">Network</a> and <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/monitoring/metrics.html#io">task I/O</a> metrics are more lightweight than the backpressure monitor and are continuously published for each running job. We can leverage those and get even more insights, not only for backpressure monitoring. The most re [...]
+
+<ul>
+ <li>
+ <p><strong><span style="color:orange">up to Flink 1.8:</span></strong> <code>outPoolUsage</code>, <code>inPoolUsage</code><br />
+An estimate on the ratio of buffers used vs. buffers available in the respective local buffer pools.
+While interpreting <code>inPoolUsage</code> in Flink 1.5 - 1.8 with credit-based flow control, please note that this only relates to floating buffers (exclusive buffers are not part of the pool).</p>
+ </li>
+ <li>
+ <p><strong><span style="color:green">Flink 1.9 and above:</span></strong> <code>outPoolUsage</code>, <code>inPoolUsage</code>, <code>floatingBuffersUsage</code>, <code>exclusiveBuffersUsage</code><br />
+An estimate on the ratio of buffers used vs. buffers available in the respective local buffer pools.
+Starting with Flink 1.9, <code>inPoolUsage</code> is the sum of <code>floatingBuffersUsage</code> and <code>exclusiveBuffersUsage</code>.</p>
+ </li>
+ <li>
+ <p><code>numRecordsOut</code>, <code>numRecordsIn</code><br />
+Each metric comes with two scopes: one scoped to the operator and one scoped to the subtask. For network monitoring, the subtask-scoped metric is relevant and shows the total number of records it has sent/received. You may need to further look into these figures to extract the number of records within a certain time span or use the equivalent <code>…PerSecond</code> metrics.</p>
+ </li>
+ <li>
+ <p><code>numBytesOut</code>, <code>numBytesInLocal</code>, <code>numBytesInRemote</code><br />
+The total number of bytes this subtask has emitted or read from a local/remote source. These are also available as meters via <code>…PerSecond</code> metrics.</p>
+ </li>
+ <li>
+ <p><code>numBuffersOut</code>, <code>numBuffersInLocal</code>, <code>numBuffersInRemote</code><br />
+Similar to <code>numBytes…</code> but counting the number of network buffers.</p>
+ </li>
+</ul>
+
+<div class="alert alert-warning">
+ <p><span class="label label-warning" style="display: inline-block"><span class="glyphicon glyphicon-warning-sign" aria-hidden="true"></span> Warning</span>
+For the sake of completeness and since they have been used in the past, we will briefly look at the <code>outputQueueLength</code> and <code>inputQueueLength</code> metrics. These are somewhat similar to the <code>[out,in]PoolUsage</code> metrics but show the number of buffers sitting in a sender subtask’s output queues and in a receiver subtask’s input queues, respectively. Reasoning about absolute numbers of buffers, however, is difficult and there i [...]
+
+ <p>Overall, <strong>we discourage the use of</strong> <code>outputQueueLength</code> <strong>and</strong> <code>inputQueueLength</code> because their interpretation highly depends on the current parallelism of the operator and the configured numbers of exclusive and floating buffers. Instead, we recommend using the various <code>*PoolUsage</code> metrics which even reveal more detailed insight.</p>
+</div>
+
+<div class="alert alert-info">
+ <p><span class="label label-info" style="display: inline-block"><span class="glyphicon glyphicon-info-sign" aria-hidden="true"></span> Note</span>
+ If you reason about buffer usage, please keep the following in mind:</p>
+
+ <ul>
+ <li>Any outgoing channel which has been used at least once will always occupy one buffer (since Flink 1.5).
+ <ul>
+ <li><strong><span style="color:orange">up to Flink 1.8:</span></strong> This buffer (even if empty!) was always counted as a backlog of 1 and thus receivers tried to reserve a floating buffer for it.</li>
+ <li><strong><span style="color:green">Flink 1.9 and above:</span></strong> A buffer is only counted in the backlog if it is ready for consumption, i.e. it is full or was flushed (see FLINK-11082)</li>
+ </ul>
+ </li>
+ <li>The receiver will only release a received buffer after deserialising the last record in it.</li>
+ </ul>
+</div>
+
+<p>The following sections make use of and combine these metrics to reason about backpressure and resource usage / efficiency with respect to throughput. A separate section will detail latency related metrics.</p>
+
+<h3 id="backpressure">Backpressure</h3>
+
+<p>Backpressure may be indicated by two different sets of metrics: (local) buffer pool usages as well as input/output queue lengths. They provide a different level of granularity but, unfortunately, none of these are exhaustive and there is room for interpretation. Because of the inherent problems with interpreting these queue lengths we will focus on the usage of input and output pools below which also provides more detail.</p>
+
+<ul>
+ <li>
+ <p><strong>If a subtask’s</strong> <code>outPoolUsage</code> <strong>is 100%</strong>, it is backpressured. Whether the subtask is already blocking or still writing records into network buffers depends on how full the buffers are, that the <code>RecordWriters</code> are currently writing into.<br />
+<span class="glyphicon glyphicon-warning-sign" aria-hidden="true" style="color:orange;"></span> This is different to what the backpressure monitor is showing!</p>
+ </li>
+ <li>
+ <p>An <code>inPoolUsage</code> of 100% means that all floating buffers are assigned to channels and eventually backpressure will be exercised upstream. These floating buffers are in either of the following conditions: they are reserved for future use on a channel due to an exclusive buffer being utilised (remote input channels always try to maintain <code>#exclusive buffers</code> credits), they are reserved for a sender’s backlog and wait for data, they [...]
+ </li>
+ <li>
+ <p><strong><span style="color:orange">up to Flink 1.8:</span></strong> Due to <a href="https://issues.apache.org/jira/browse/FLINK-11082">FLINK-11082</a>, an <code>inPoolUsage</code> of 100% is quite common even in normal situations.</p>
+ </li>
+ <li>
+ <p><strong><span style="color:green">Flink 1.9 and above:</span></strong> If <code>inPoolUsage</code> is constantly around 100%, this is a strong indicator for exercising backpressure upstream.</p>
+ </li>
+</ul>
+
+<p>The following table summarises all combinations and their interpretation. Bear in mind, though, that backpressure may be minor or temporary (no need to look into it), on particular channels only, or caused by other JVM processes on a particular TaskManager, such as GC, synchronisation, I/O, resource shortage, instead of a specific subtask.</p>
+
+<center>
+<table class="tg">
+ <tr>
+ <th></th>
+ <th class="tg-center"><code>outPoolUsage</code> low</th>
+ <th class="tg-center"><code>outPoolUsage</code> high</th>
+ </tr>
+ <tr>
+ <th class="tg-top"><code>inPoolUsage</code> low</th>
+ <td class="tg-topcenter">
+ <span class="glyphicon glyphicon-ok-sign" aria-hidden="true" style="color:green;font-size:1.5em;"></span></td>
+ <td class="tg-topcenter">
+ <span class="glyphicon glyphicon-warning-sign" aria-hidden="true" style="color:orange;font-size:1.5em;"></span><br />
+ (backpressured, temporary situation: upstream is not backpressured yet or not anymore)</td>
+ </tr>
+ <tr>
+ <th class="tg-top" rowspan="2">
+ <code>inPoolUsage</code> high<br />
+ (<strong><span style="color:green">Flink 1.9+</span></strong>)</th>
+ <td class="tg-topcenter">
+ if all upstream tasks’<code>outPoolUsage</code> are low: <span class="glyphicon glyphicon-warning-sign" aria-hidden="true" style="color:orange;font-size:1.5em;"></span><br />
+ (may eventually cause backpressure)</td>
+ <td class="tg-topcenter" rowspan="2">
+ <span class="glyphicon glyphicon-remove-sign" aria-hidden="true" style="color:red;font-size:1.5em;"></span><br />
+ (backpressured by downstream task(s) or network, probably forwarding backpressure upstream)</td>
+ </tr>
+ <tr>
+ <td class="tg-topcenter">if any upstream task’s<code>outPoolUsage</code> is high: <span class="glyphicon glyphicon-remove-sign" aria-hidden="true" style="color:red;font-size:1.5em;"></span><br />
+ (may exercise backpressure upstream and may be the source of backpressure)</td>
+ </tr>
+</table>
+</center>
+
+<p><br />
+We may even reason more about the cause of backpressure by looking at the network metrics of the subtasks of two consecutive tasks:</p>
+
+<ul>
+ <li>If all subtasks of the receiver task have low <code>inPoolUsage</code> values and any upstream subtask’s <code>outPoolUsage</code> is high, then there may be a network bottleneck causing backpressure.
+Since network is a shared resource among all subtasks of a TaskManager, this may not directly originate from this subtask, but rather from various concurrent operations, e.g. checkpoints, other streams, external connections, or other TaskManagers/processes on the same machine.</li>
+</ul>
+
+<p>Backpressure can also be caused by all parallel instances of a task or by a single task instance. The first usually happens because the task is performing some time consuming operation that applies to all input partitions. The latter is usually the result of some kind of skew, either data skew or resource availability/allocation skew. In either case, you can find some hints on how to handle such situations in the <a href="#span-classlabel-label-info-styledisplay-inline-b [...]
+
+<div class="alert alert-info">
+ <h3 class="no_toc" id="span-classglyphicon-glyphicon-info-sign-aria-hiddentruespan-flink-19-and-above"><span class="glyphicon glyphicon-info-sign" aria-hidden="true"></span> Flink 1.9 and above</h3>
+
+ <ul>
+ <li>If <code>floatingBuffersUsage</code> is not 100%, it is unlikely that there is backpressure. If it is 100% and any upstream task is backpressured, it suggests that this input is exercising backpressure on either a single, some or all input channels. To differentiate between those three situations you can use <code>exclusiveBuffersUsage</code>:
+ <ul>
+ <li>Assuming that <code>floatingBuffersUsage</code> is around 100%, the higher the <code>exclusiveBuffersUsage</code> the more input channels are backpressured. In an extreme case of <code>exclusiveBuffersUsage</code> being close to 100%, it means that all channels are backpressured.</li>
+ </ul>
+ </li>
+ </ul>
+
+ <p><br />
+The relation between <code>exclusiveBuffersUsage</code>, <code>floatingBuffersUsage</code>, and the upstream tasks’ <code>outPoolUsage</code> is summarised in the following table and extends on the table above with <code>inPoolUsage = floatingBuffersUsage + exclusiveBuffersUsage</code>:</p>
+
+ <center>
+<table class="tg">
+ <tr>
+ <th></th>
+ <th><code>exclusiveBuffersUsage</code> low</th>
+ <th><code>exclusiveBuffersUsage</code> high</th>
+ </tr>
+ <tr>
+ <th class="tg-top" style="min-width:33%;">
+ <code>floatingBuffersUsage</code> low +<br />
+ <em>all</em> upstream <code>outPoolUsage</code> low</th>
+ <td class="tg-center"><span class="glyphicon glyphicon-ok-sign" aria-hidden="true" style="color:green;font-size:1.5em;"></span></td>
+ <td class="tg-center">-<sup>3</sup></td>
+ </tr>
+ <tr>
+ <th class="tg-top" style="min-width:33%;">
+ <code>floatingBuffersUsage</code> low +<br />
+ <em>any</em> upstream <code>outPoolUsage</code> high</th>
+ <td class="tg-center">
+ <span class="glyphicon glyphicon-remove-sign" aria-hidden="true" style="color:red;font-size:1.5em;"></span><br />
+ (potential network bottleneck)</td>
+ <td class="tg-center">-<sup>3</sup></td>
+ </tr>
+ <tr>
+ <th class="tg-top" style="min-width:33%;">
+ <code>floatingBuffersUsage</code> high +<br />
+ <em>all</em> upstream <code>outPoolUsage</code> low</th>
+ <td class="tg-center">
+ <span class="glyphicon glyphicon-warning-sign" aria-hidden="true" style="color:orange;font-size:1.5em;"></span><br />
+ (backpressure eventually appears on only some of the input channels)</td>
+ <td class="tg-center">
+ <span class="glyphicon glyphicon-warning-sign" aria-hidden="true" style="color:orange;font-size:1.5em;"></span><br />
+ (backpressure eventually appears on most or all of the input channels)</td>
+ </tr>
+ <tr>
+ <th class="tg-top" style="min-width:33%;">
+ <code>floatingBuffersUsage</code> high +<br />
+ any upstream <code>outPoolUsage</code> high</th>
+ <td class="tg-center">
+ <span class="glyphicon glyphicon-remove-sign" aria-hidden="true" style="color:red;font-size:1.5em;"></span><br />
+ (backpressure on only some of the input channels)</td>
+ <td class="tg-center">
+ <span class="glyphicon glyphicon-remove-sign" aria-hidden="true" style="color:red;font-size:1.5em;"></span><br />
+ (backpressure on most or all of the input channels)</td>
+ </tr>
+</table>
+</center>
+
+ <p><sup>3</sup> this should not happen</p>
+
+</div>
+
+<h3 id="resource-usage--throughput">Resource Usage / Throughput</h3>
+
+<p>Besides the obvious use of each individual metric mentioned above, there are also a few combinations providing useful insight into what is happening in the network stack:</p>
+
+<ul>
+ <li>
+ <p>Low throughput with frequent <code>outPoolUsage</code> values around 100% but low <code>inPoolUsage</code> on all receivers is an indicator that the round-trip-time of our credit-notification (depends on your network’s latency) is too high for the default number of exclusive buffers to make use of your bandwidth. Consider increasing the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/ops/config.html#taskmanager-network-mem [...]
+ </li>
+ <li>
+ <p>Combining <code>numRecordsOut</code> and <code>numBytesOut</code> helps identifying average serialised record sizes which supports you in capacity planning for peak scenarios.</p>
+ </li>
+ <li>
+ <p>If you want to reason about buffer fill rates and the influence of the output flusher, you may combine <code>numBytesInRemote</code> with <code>numBuffersInRemote</code>. When tuning for throughput (and not latency!), low buffer fill rates may indicate reduced network efficiency. In such cases, consider increasing the buffer timeout.
+Please note that, as of Flink 1.8 and 1.9, <code>numBuffersOut</code> only increases for buffers getting full or for an event cutting off a buffer (e.g. a checkpoint barrier) and may lag behind. Please also note that reasoning about buffer fill rates on local channels is unnecessary since buffering is an optimisation technique for remote channels with limited effect on local channels.</p>
+ </li>
+ <li>
+ <p>You may also separate local from remote traffic using numBytesInLocal and numBytesInRemote but in most cases this is unnecessary.</p>
+ </li>
+</ul>
+
+<div class="alert alert-info">
+ <h3 class="no_toc" id="span-classglyphicon-glyphicon-info-sign-aria-hiddentruespan-what-to-do-with-backpressure"><span class="glyphicon glyphicon-info-sign" aria-hidden="true"></span> What to do with Backpressure?</h3>
+
+ <p>Assuming that you identified where the source of backpressure — a bottleneck — is located, the next step is to analyse why this is happening. Below, we list some potential causes of backpressure from the more basic to the more complex ones. We recommend to check the basic causes first, before diving deeper on the more complex ones and potentially drawing false conclusions.</p>
+
+ <p>Please also recall that backpressure might be temporary and the result of a load spike, checkpointing, or a job restart with a data backlog waiting to be processed. In that case, you can often just ignore it. Alternatively, keep in mind that the process of analysing and solving the issue can be affected by the intermittent nature of your bottleneck. Having said that, here are a couple of things to check.</p>
+
+ <h4 id="system-resources">System Resources</h4>
+
+ <p>Firstly, you should check the incriminated machines’ basic resource usage like CPU, network, or disk I/O. If some resource is fully or heavily utilised you can do one of the following:</p>
+
+ <ol>
+ <li>Try to optimise your code. Code profilers are helpful in this case.</li>
+ <li>Tune Flink for that specific resource.</li>
+ <li>Scale out by increasing the parallelism and/or increasing the number of machines in the cluster.</li>
+ </ol>
+
+ <h4 id="garbage-collection">Garbage Collection</h4>
+
+ <p>Oftentimes, performance issues arise from long GC pauses. You can verify whether you are in such a situation by either printing debug GC logs (via -<code>XX:+PrintGCDetails</code>) or by using some memory/GC profilers. Since dealing with GC issues is highly application-dependent and independent of Flink, we will not go into details here (<a href="https://docs.oracle.com/javase/8/docs/technotes/guides/vm/gctuning/index.html">Oracle’s Garbage Collecti [...]
+
+ <h4 id="cputhread-bottleneck">CPU/Thread Bottleneck</h4>
+
+ <p>Sometimes a CPU bottleneck might not be visible at first glance if one or a couple of threads are causing the CPU bottleneck while the CPU usage of the overall machine remains relatively low. For instance, a single CPU-bottlenecked thread on a 48-core machine would result in only 2% CPU use. Consider using code profilers for this as they can identify hot threads by showing each threads’ CPU usage, for example.</p>
+
+ <h4 id="thread-contention">Thread Contention</h4>
+
+ <p>Similarly to the CPU/thread bottleneck issue above, a subtask may be bottlenecked due to high thread contention on shared resources. Again, CPU profilers are your best friend here! Consider looking for synchronisation overhead / lock contention in user code — although adding synchronisation in user code should be avoided and may even be dangerous! Also consider investigating shared system resources. The default JVM’s SSL implementation, for example, can become contented around [...]
+
+ <h4 id="load-imbalance">Load Imbalance</h4>
+
+ <p>If your bottleneck is caused by data skew, you can try to remove it or mitigate its impact by changing the data partitioning to separate heavy keys or by implementing local/pre-aggregation.</p>
+
+ <p><br />
+This list is far from exhaustive. Generally, in order to reduce a bottleneck and thus backpressure, first analyse where it is happening and then find out why. The best place to start reasoning about the “why” is by checking what resources are fully utilised.</p>
+</div>
+
+<h3 id="latency-tracking">Latency Tracking</h3>
+
+<p>Tracking latencies at the various locations they may occur is a topic of its own. In this section, we will focus on the time records wait inside Flink’s network stack — including the system’s network connections. In low throughput scenarios, these latencies are influenced directly by the output flusher via the buffer timeout parameter or indirectly by any application code latencies. When processing a record takes longer than expected or when (multiple) timers fire at the same ti [...]
+
+<p>Flink offers some support for <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/monitoring/metrics.html#latency-tracking">tracking the latency</a> of records passing through the system (outside of user code). However, this is disabled by default (see below why!) and must be enabled by setting a latency tracking interval either in Flink’s <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/ops/config.html#metrics-l [...]
+
+<ul>
+ <li><code>single</code>: one histogram for each operator subtask</li>
+ <li><code>operator</code> (default): one histogram for each combination of source task and operator subtask</li>
+ <li><code>subtask</code>: one histogram for each combination of source subtask and operator subtask (quadratic in the parallelism!)</li>
+</ul>
+
+<p>These metrics are collected through special “latency markers”: each source subtask will periodically emit a special record containing the timestamp of its creation. The latency markers then flow alongside normal records while not overtaking them on the wire or inside a buffer queue. However, <em>a latency marker does not enter application logic</em> and is overtaking records there. Latency markers therefore only measure the waiting time between the user code and not [...]
+
+<p>Since <code>LatencyMarkers</code> sit in network buffers just like normal records, they will also wait for the buffer to be full or flushed due to buffer timeouts. When a channel is on high load, there is no added latency by the network buffering data. However, as soon as one channel is under low load, records and latency markers will experience an expected average delay of at most <code>buffer_timeout / 2</code>. This delay will add to each network conne [...]
+
+<p>By looking at the exposed latency tracking metrics for each subtask, for example at the 95th percentile, you should nevertheless be able to identify subtasks which are adding substantially to the overall source-to-sink latency and continue with optimising there.</p>
+
+<div class="alert alert-info">
+ <p><span class="label label-info" style="display: inline-block"><span class="glyphicon glyphicon-info-sign" aria-hidden="true"></span> Note</span>
+Flink’s latency markers assume that the clocks on all machines in the cluster are in sync. We recommend setting up an automated clock synchronisation service (like NTP) to avoid false latency results.</p>
+</div>
+
+<div class="alert alert-warning">
+ <p><span class="label label-warning" style="display: inline-block"><span class="glyphicon glyphicon-warning-sign" aria-hidden="true"></span> Warning</span>
+Enabling latency metrics can significantly impact the performance of the cluster (in particular for <code>subtask</code> granularity) due to the sheer amount of metrics being added as well as the use of histograms which are quite expensive to maintain. It is highly recommended to only use them for debugging purposes.</p>
+</div>
+
+<h2 id="conclusion">Conclusion</h2>
+
+<p>In the previous sections we discussed how to monitor Flink’s network stack which primarily involves identifying backpressure: where it occurs, where it originates from, and (potentially) why it occurs. This can be executed in two ways: for simple cases and debugging sessions by using the backpressure monitor; for continuous monitoring, more in-depth analysis, and less runtime overhead by using Flink’s task and network stack metrics. Backpressure can be caused by the network laye [...]
+
+<p>Stay tuned for the third blog post in the series of network stack posts that will focus on tuning techniques and anti-patterns to avoid.</p>
+
+</description>
+<pubDate>Tue, 23 Jul 2019 17:30:00 +0200</pubDate>
+<link>https://flink.apache.org/2019/07/23/flink-network-stack-2.html</link>
+<guid isPermaLink="true">/2019/07/23/flink-network-stack-2.html</guid>
+</item>
+
+<item>
+<title>Apache Flink 1.8.1 Released</title>
+<description><p>The Apache Flink community released the first bugfix version of the Apache Flink 1.8 series.</p>
+
+<p>This release includes more than 40 fixes and minor improvements for Flink 1.8.1. The list below includes a detailed list of all improvements, sub-tasks and bug fixes.</p>
+
+<p>We highly recommend all users to upgrade to Flink 1.8.1.</p>
+
+<p>Updated Maven dependencies:</p>
+
+<div class="highlight"><pre><code class="language-xml"><span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-java<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.8.1<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span>
+<span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-streaming-java_2.11<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.8.1<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span>
+<span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-clients_2.11<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.8.1<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span></code></pre></div>
+
+<p>You can find the binaries on the updated <a href="/downloads.html">Downloads page</a>.</p>
+
+<p>List of resolved issues:</p>
+
+<h2> Sub-task
+</h2>
+<ul>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10921">FLINK-10921</a>] - Prioritize shard consumers in Kinesis Consumer by event time
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12617">FLINK-12617</a>] - StandaloneJobClusterEntrypoint should default to random JobID for non-HA setups
+</li>
+</ul>
+
+<h2> Bug
+</h2>
+<ul>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-9445">FLINK-9445</a>] - scala-shell uses plain java command
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10455">FLINK-10455</a>] - Potential Kafka producer leak in case of failures
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10941">FLINK-10941</a>] - Slots prematurely released which still contain unconsumed data
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11059">FLINK-11059</a>] - JobMaster may continue using an invalid slot if releasing idle slot meet a timeout
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11107">FLINK-11107</a>] - Avoid memory stateBackend to create arbitrary folders under HA path when no checkpoint path configured
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11897">FLINK-11897</a>] - ExecutionGraphSuspendTest does not wait for all tasks to be submitted
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11915">FLINK-11915</a>] - DataInputViewStream skip returns wrong value
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11987">FLINK-11987</a>] - Kafka producer occasionally throws NullpointerException
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12009">FLINK-12009</a>] - Wrong check message about heartbeat interval for HeartbeatServices
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12042">FLINK-12042</a>] - RocksDBStateBackend mistakenly uses default filesystem
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12112">FLINK-12112</a>] - AbstractTaskManagerProcessFailureRecoveryTest process output logging does not work properly
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12132">FLINK-12132</a>] - The example in /docs/ops/deployment/yarn_setup.md should be updated due to the change FLINK-2021
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12184">FLINK-12184</a>] - HistoryServerArchiveFetcher isn&#39;t compatible with old version
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12219">FLINK-12219</a>] - Yarn application can&#39;t stop when flink job failed in per-job yarn cluster mode
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12247">FLINK-12247</a>] - fix NPE when writing an archive file to a FileSystem
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12260">FLINK-12260</a>] - Slot allocation failure by taskmanager registration timeout and race
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12296">FLINK-12296</a>] - Data loss silently in RocksDBStateBackend when more than one operator(has states) chained in a single task
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12297">FLINK-12297</a>] - Make ClosureCleaner recursive
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12301">FLINK-12301</a>] - Scala value classes inside case classes cannot be serialized anymore in Flink 1.8.0
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12342">FLINK-12342</a>] - Yarn Resource Manager Acquires Too Many Containers
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12375">FLINK-12375</a>] - flink-container job jar does not have read permissions
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12416">FLINK-12416</a>] - Docker build script fails on symlink creation ln -s
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12544">FLINK-12544</a>] - Deadlock while releasing memory and requesting segment concurrent in SpillableSubpartition
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12547">FLINK-12547</a>] - Deadlock when the task thread downloads jars using BlobClient
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12646">FLINK-12646</a>] - Use reserved IP as unrouteable IP in RestClientTest
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12688">FLINK-12688</a>] - Make serializer lazy initialization thread safe in StateDescriptor
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12740">FLINK-12740</a>] - SpillableSubpartitionTest deadlocks on Travis
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12835">FLINK-12835</a>] - Time conversion is wrong in ManualClock
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12863">FLINK-12863</a>] - Race condition between slot offerings and AllocatedSlotReport
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12865">FLINK-12865</a>] - State inconsistency between RM and TM on the slot status
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12871">FLINK-12871</a>] - Wrong SSL setup examples in docs
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12895">FLINK-12895</a>] - TaskManagerProcessFailureBatchRecoveryITCase.testTaskManagerProcessFailure failed on travis
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12896">FLINK-12896</a>] - TaskCheckpointStatisticDetailsHandler uses wrong value for JobID when archiving
+</li>
+</ul>
+
+<h2> Improvement
+</h2>
+<ul>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11126">FLINK-11126</a>] - Filter out AMRMToken in the TaskManager credentials
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12137">FLINK-12137</a>] - Add more proper explanation on flink streaming connectors
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12169">FLINK-12169</a>] - Improve Javadoc of MessageAcknowledgingSourceBase
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12378">FLINK-12378</a>] - Consolidate FileSystem Documentation
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12391">FLINK-12391</a>] - Add timeout to transfer.sh
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12539">FLINK-12539</a>] - StreamingFileSink: Make the class extendable to customize for different usecases
+</li>
+</ul>
+
+<h2> Test
+</h2>
+<ul>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12350">FLINK-12350</a>] - RocksDBStateBackendTest doesn&#39;t cover the incremental checkpoint code path
+</li>
+</ul>
+
+<h2> Task
+</h2>
+<ul>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-12460">FLINK-12460</a>] - Change taskmanager.tmp.dirs to io.tmp.dirs in configuration docs
+</li>
+</ul>
+
+</description>
+<pubDate>Tue, 02 Jul 2019 14:00:00 +0200</pubDate>
+<link>https://flink.apache.org/news/2019/07/02/release-1.8.1.html</link>
+<guid isPermaLink="true">/news/2019/07/02/release-1.8.1.html</guid>
+</item>
+
+<item>
+<title>A Practical Guide to Broadcast State in Apache Flink</title>
+<description><p>Since version 1.5.0, Apache Flink features a new type of state which is called Broadcast State. In this post, we explain what Broadcast State is, and show an example of how it can be applied to an application that evaluates dynamic patterns on an event stream. We walk you through the processing steps and the source code to implement this application in practice.</p>
+
+<h2 id="what-is-broadcast-state">What is Broadcast State?</h2>
+
+<p>The Broadcast State can be used to combine and jointly process two streams of events in a specific way. The events of the first stream are broadcasted to all parallel instances of an operator, which maintains them as state. The events of the other stream are not broadcasted but sent to individual instances of the same operator and processed together with the events of the broadcasted stream.
+The new broadcast state is a natural fit for applications that need to join a low-throughput and a high-throughput stream or need to dynamically update their processing logic. We will use a concrete example of the latter use case to explain the broadcast state and show its API in more detail in the remainder of this post.</p>
+
+<h2 id="dynamic-pattern-evaluation-with-broadcast-state">Dynamic Pattern Evaluation with Broadcast State</h2>
+
+<p>Imagine an e-commerce website that captures the interactions of all users as a stream of user actions. The company that operates the website is interested in analyzing the interactions to increase revenue, improve the user experience, and detect and prevent malicious behavior.
+The website implements a streaming application that detects a pattern on the stream of user events. However, the company wants to avoid modifying and redeploying the application every time the pattern changes. Instead, the application ingests a second stream of patterns and updates its active pattern when it receives a new pattern from the pattern stream. In the following, we discuss this application step-by-step and show how it leverages the broadcast state feature in Apache Flink.</p>
+
+<center>
+<img src="/img/blog/broadcastState/fig1.png" width="600px" alt="Broadcast State in Apache Flink." />
+</center>
+<p><br /></p>
+
+<p>Our example application ingests two data streams. The first stream provides user actions on the website and is illustrated on the top left side of the above figure. A user interaction event consists of the type of the action (user login, user logout, add to cart, or complete payment) and the id of the user, which is encoded by color. The user action event stream in our illustration contains a logout action of User 1001 followed by a payment-complete event for User 1003, and an “ [...]
+
+<p>The second stream provides action patterns that the application will evaluate. A pattern consists of two consecutive actions. In the figure above, the pattern stream contains the following two:</p>
+
+<ul>
+ <li>Pattern #1: A user logs in and immediately logs out without browsing additional pages on the e-commerce website.</li>
+ <li>Pattern #2: A user adds an item to the shopping cart and logs out without completing the purchase.</li>
+</ul>
+
+<p>Such patterns help a business in better analyzing user behavior, detecting malicious actions, and improving the website experience. For example, in the case of items being added to a shopping cart with no follow up purchase, the website team can take appropriate actions to understand better the reasons why users don’t complete a purchase and initiate specific programs to improve the website conversion (such as providing discount codes, limited free shipping offers etc.)</p>
+
+<p>On the right-hand side, the figure shows three parallel tasks of an operator that ingest the pattern and user action streams, evaluate the patterns on the action stream, and emit pattern matches downstream. For the sake of simplicity, the operator in our example only evaluates a single pattern with exactly two subsequent actions. The currently active pattern is replaced when a new pattern is received from the pattern stream. In principle, the operator could also be implemented t [...]
+
+<p>We will describe how the pattern matching application processes the user action and pattern streams.</p>
+
+<center>
+<img src="/img/blog/broadcastState/fig2.png" width="600px" alt="Broadcast State in Apache Flink." />
+</center>
+<p><br /></p>
+
+<p>First a pattern is sent to the operator. The pattern is broadcasted to all three parallel tasks of the operator. The tasks store the pattern in their broadcast state. Since the broadcast state should only be updated using broadcasted data, the state of all tasks is always expected to be the same.</p>
+
+<center>
+<img src="/img/blog/broadcastState/fig3.png" width="600px" alt="Broadcast State in Apache Flink." />
+</center>
+<p><br /></p>
+
+<p>Next, the first user actions are partitioned on the user id and shipped to the operator tasks. The partitioning ensures that all actions of the same user are processed by the same task. The figure above shows the state of the application after the first pattern and the first three action events were consumed by the operator tasks.</p>
+
+<p>When a task receives a new user action, it evaluates the currently active pattern by looking at the user’s latest and previous actions. For each user, the operator stores the previous action in the keyed state. Since the tasks in the figure above only received a single action for each user so far (we just started the application), the pattern does not need to be evaluated. Finally, the previous action in the user’s keyed state is updated to the latest action, to be able to look [...]
+
+<center>
+<img src="/img/blog/broadcastState/fig4.png" width="600px" alt="Broadcast State in Apache Flink." />
+</center>
+<p><br /></p>
+
+<p>After the first three actions are processed, the next event, the logout action of User 1001, is shipped to the task that processes the events of User 1001. When the task receives the actions, it looks up the current pattern from the broadcast state and the previous action of User 1001. Since the pattern matches both actions, the task emits a pattern match event. Finally, the task updates its keyed state by overriding the previous event with the latest action.</p>
+
+<center>
+<img src="/img/blog/broadcastState/fig5.png" width="600px" alt="Broadcast State in Apache Flink." />
+</center>
+<p><br /></p>
+
+<p>When a new pattern arrives in the pattern stream, it is broadcasted to all tasks and each task updates its broadcast state by replacing the current pattern with the new one.</p>
+
+<center>
+<img src="/img/blog/broadcastState/fig6.png" width="600px" alt="Broadcast State in Apache Flink." />
+</center>
+<p><br /></p>
+
+<p>Once the broadcast state is updated with a new pattern, the matching logic continues as before, i.e., user action events are partitioned by key and evaluated by the responsible task.</p>
+
+<h2 id="how-to-implement-an-application-with-broadcast-state">How to Implement an Application with Broadcast State?</h2>
+
+<p>Until now, we conceptually discussed the application and explained how it uses broadcast state to evaluate dynamic patterns over event streams. Next, we’ll show how to implement the example application with Flink’s DataStream API and the broadcast state feature.</p>
+
+<p>Let’s start with the input data of the application. We have two data streams, actions, and patterns. At this point, we don’t really care where the streams come from. The streams could be ingested from Apache Kafka or Kinesis or any other system. Action and Pattern are Pojos with two fields each:</p>
+
+<div class="highlight"><pre><code class="language-java"><span class="n">DataStream</span><span class="o">&lt;</span><span class="n">Action</span><span class="o">&gt;</span> <span class="n">actions</span> <span class="o">=</span> <span class="o">???</span>
+<span class="n">DataStream</span><span class="o">&lt;</span><span class="n">Pattern</span><span class="o">&gt;</span> <span class="n">patterns</span> <span class="o">=</span> <span class="o">???</span></code></pre></div>
+
+<p><code>Action</code> and <code>Pattern</code> are Pojos with two fields each:</p>
+
+<ul>
+ <li>
+ <p><code>Action: Long userId, String action</code></p>
+ </li>
+ <li>
+ <p><code>Pattern: String firstAction, String secondAction</code></p>
+ </li>
+</ul>
+
+<p>As a first step, we key the action stream on the <code>userId</code> attribute.</p>
+
+<div class="highlight"><pre><code class="language-java"><span class="n">KeyedStream</span><span class="o">&lt;</span><span class="n">Action</span><span class="o">,</span> <span class="n">Long</span><span class="o">&gt;</span> <span class="n">actionsByUser</span> <span class="o"& [...]
+ <span class="o">.</span><span class="na">keyBy</span><span class="o">((</span><span class="n">KeySelector</span><span class="o">&lt;</span><span class="n">Action</span><span class="o">,</span> <span class="n">Long</span><span class="o">&gt;)</span> <span class="n">act [...]
+
+<p>Next, we prepare the broadcast state. Broadcast state is always represented as <code>MapState</code>, the most versatile state primitive that Flink provides.</p>
+
+<div class="highlight"><pre><code class="language-java"><span class="n">MapStateDescriptor</span><span class="o">&lt;</span><span class="n">Void</span><span class="o">,</span> <span class="n">Pattern</span><span class="o">&gt;</span> <span class="n">bcStateDescriptor</span> <span class=&q [...]
+ <span class="k">new</span> <span class="n">MapStateDescriptor</span><span class="o">&lt;&gt;(</span><span class="s">&quot;patterns&quot;</span><span class="o">,</span> <span class="n">Types</span><span class="o">.</span><span class="na">VOID</span><span class="o">,</span> < [...]
+
+<p>Since our application only evaluates and stores a single <code>Pattern</code> at a time, we configure the broadcast state as a <code>MapState</code> with key type <code>Void</code> and value type <code>Pattern</code>. The <code>Pattern</code> is always stored in the <code>MapState</code> with <code>null</code> as key.</p>
+
+<div class="highlight"><pre><code class="language-java"><span class="n">BroadcastStream</span><span class="o">&lt;</span><span class="n">Pattern</span><span class="o">&gt;</span> <span class="n">bcedPatterns</span> <span class="o">=</span> <span class="n">patterns</span><span class=" [...]
+<p>Using the <code>MapStateDescriptor</code> for the broadcast state, we apply the <code>broadcast()</code> transformation on the patterns stream and receive a <code>BroadcastStream bcedPatterns</code>.</p>
+
+<div class="highlight"><pre><code class="language-java"><span class="n">DataStream</span><span class="o">&lt;</span><span class="n">Tuple2</span><span class="o">&lt;</span><span class="n">Long</span><span class="o">,</span> <span class="n">Pattern</span><span class="o">&g [...]
+ <span class="o">.</span><span class="na">connect</span><span class="o">(</span><span class="n">bcedPatterns</span><span class="o">)</span>
+ <span class="o">.</span><span class="na">process</span><span class="o">(</span><span class="k">new</span> <span class="nf">PatternEvaluator</span><span class="o">());</span></code></pre></div>
+
+<p>After we obtained the keyed <code>actionsByUser</code> stream and the broadcasted <code>bcedPatterns</code> stream, we <code>connect()</code> both streams and apply a <code>PatternEvaluator</code> on the connected streams. <code>PatternEvaluator</code> is a custom function that implements the <code>KeyedBroadcastProcessFunction</code> interface. It applies the pattern matching logic that we discussed before [...]
+
+<div class="highlight"><pre><code class="language-java"><span class="kd">public</span> <span class="kd">static</span> <span class="kd">class</span> <span class="nc">PatternEvaluator</span>
+ <span class="kd">extends</span> <span class="n">KeyedBroadcastProcessFunction</span><span class="o">&lt;</span><span class="n">Long</span><span class="o">,</span> <span class="n">Action</span><span class="o">,</span> <span class="n">Pattern</span><span class="o">,</span> <span class [...]
+
+ <span class="c1">// handle for keyed state (per user)</span>
+ <span class="n">ValueState</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">prevActionState</span><span class="o">;</span>
+ <span class="c1">// broadcast state descriptor</span>
+ <span class="n">MapStateDescriptor</span><span class="o">&lt;</span><span class="n">Void</span><span class="o">,</span> <span class="n">Pattern</span><span class="o">&gt;</span> <span class="n">patternDesc</span><span class="o">;</span>
+
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="nf">open</span><span class="o">(</span><span class="n">Configuration</span> <span class="n">conf</span><span class="o">)</span> <span class="o">{</span>
+ <span class="c1">// initialize keyed state</span>
+ <span class="n">prevActionState</span> <span class="o">=</span> <span class="n">getRuntimeContext</span><span class="o">().</span><span class="na">getState</span><span class="o">(</span>
+ <span class="k">new</span> <span class="n">ValueStateDescriptor</span><span class="o">&lt;&gt;(</span><span class="s">&quot;lastAction&quot;</span><span class="o">,</span> <span class="n">Types</span><span class="o">.</span><span class="na">STRING</span><span class="o">));</ [...]
+ <span class="n">patternDesc</span> <span class="o">=</span>
+ <span class="k">new</span> <span class="n">MapStateDescriptor</span><span class="o">&lt;&gt;(</span><span class="s">&quot;patterns&quot;</span><span class="o">,</span> <span class="n">Types</span><span class="o">.</span><span class="na">VOID</span><span class="o">,</span> [...]
+ <span class="o">}</span>
+
+ <span class="cm">/**</span>
+<span class="cm"> * Called for each user action.</span>
+<span class="cm"> * Evaluates the current pattern against the previous and</span>
+<span class="cm"> * current action of the user.</span>
+<span class="cm"> */</span>
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="nf">processElement</span><span class="o">(</span>
+ <span class="n">Action</span> <span class="n">action</span><span class="o">,</span>
+ <span class="n">ReadOnlyContext</span> <span class="n">ctx</span><span class="o">,</span>
+ <span class="n">Collector</span><span class="o">&lt;</span><span class="n">Tuple2</span><span class="o">&lt;</span><span class="n">Long</span><span class="o">,</span> <span class="n">Pattern</span><span class="o">&gt;&gt;</span> <span class="n">out</span><span class=&qu [...]
+ <span class="c1">// get current pattern from broadcast state</span>
+ <span class="n">Pattern</span> <span class="n">pattern</span> <span class="o">=</span> <span class="n">ctx</span>
+ <span class="o">.</span><span class="na">getBroadcastState</span><span class="o">(</span><span class="k">this</span><span class="o">.</span><span class="na">patternDesc</span><span class="o">)</span>
+ <span class="c1">// access MapState with null as VOID default value</span>
+ <span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="kc">null</span><span class="o">);</span>
+ <span class="c1">// get previous action of current user from keyed state</span>
+ <span class="n">String</span> <span class="n">prevAction</span> <span class="o">=</span> <span class="n">prevActionState</span><span class="o">.</span><span class="na">value</span><span class="o">();</span>
+ <span class="k">if</span> <span class="o">(</span><span class="n">pattern</span> <span class="o">!=</span> <span class="kc">null</span> <span class="o">&amp;&amp;</span> <span class="n">prevAction</span> <span class="o">!=</span> <span class="kc">null</span><span class="o&qu [...]
+ <span class="c1">// user had an action before, check if pattern matches</span>
+ <span class="k">if</span> <span class="o">(</span><span class="n">pattern</span><span class="o">.</span><span class="na">firstAction</span><span class="o">.</span><span class="na">equals</span><span class="o">(</span><span class="n">prevAction</span><span class="o">)</s [...]
+ <span class="n">pattern</span><span class="o">.</span><span class="na">secondAction</span><span class="o">.</span><span class="na">equals</span><span class="o">(</span><span class="n">action</span><span class="o">.</span><span class="na">action</span><span class="o">)) [...]
+ <span class="c1">// MATCH</span>
+ <span class="n">out</span><span class="o">.</span><span class="na">collect</span><span class="o">(</span><span class="k">new</span> <span class="n">Tuple2</span><span class="o">&lt;&gt;(</span><span class="n">ctx</span><span class="o">.</span><span class="na">get [...]
+ <span class="o">}</span>
+ <span class="o">}</span>
+ <span class="c1">// update keyed state and remember action for next pattern evaluation</span>
+ <span class="n">prevActionState</span><span class="o">.</span><span class="na">update</span><span class="o">(</span><span class="n">action</span><span class="o">.</span><span class="na">action</span><span class="o">);</span>
+ <span class="o">}</span>
+
+ <span class="cm">/**</span>
+<span class="cm"> * Called for each new pattern.</span>
+<span class="cm"> * Overwrites the current pattern with the new pattern.</span>
+<span class="cm"> */</span>
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="nf">processBroadcastElement</span><span class="o">(</span>
+ <span class="n">Pattern</span> <span class="n">pattern</span><span class="o">,</span>
+ <span class="n">Context</span> <span class="n">ctx</span><span class="o">,</span>
+ <span class="n">Collector</span><span class="o">&lt;</span><span class="n">Tuple2</span><span class="o">&lt;</span><span class="n">Long</span><span class="o">,</span> <span class="n">Pattern</span><span class="o">&gt;&gt;</span> <span class="n">out</span><span class=&qu [...]
+ <span class="c1">// store the new pattern by updating the broadcast state</span>
+ <span class="n">BroadcastState</span><span class="o">&lt;</span><span class="n">Void</span><span class="o">,</span> <span class="n">Pattern</span><span class="o">&gt;</span> <span class="n">bcState</span> <span class="o">=</span> <span class="n">ctx</span><span class="o" [...]
+ <span class="c1">// storing in MapState with null as VOID default value</span>
+ <span class="n">bcState</span><span class="o">.</span><span class="na">put</span><span class="o">(</span><span class="kc">null</span><span class="o">,</span> <span class="n">pattern</span><span class="o">);</span>
+ <span class="o">}</span>
+<span class="o">}</span></code></pre></div>
+
+<p>The <code>KeyedBroadcastProcessFunction</code> interface provides three methods to process records and emit results.</p>
+
+<ul>
+ <li><code>processBroadcastElement()</code> is called for each record of the broadcasted stream. In our <code>PatternEvaluator</code> function, we simply put the received <code>Pattern</code> record in to the broadcast state using the <code>null</code> key (remember, we only store a single pattern in the <code>MapState</code>).</li>
+ <li><code>processElement()</code> is called for each record of the keyed stream. It provides read-only access to the broadcast state to prevent modification that result in different broadcast states across the parallel instances of the function. The <code>processElement()</code> method of the <code>PatternEvaluator</code> retrieves the current pattern from the broadcast state and the previous action of the user from the keyed state. If both are [...]
+ <li><code>onTimer()</code> is called when a previously registered timer fires. Timers can be registered in the <code>processElement</code> method and are used to perform computations or to clean up state in the future. We did not implement this method in our example to keep the code concise. However, it could be used to remove the last action of a user when the user was not active for a certain period of time to avoid growing state due to inactive users.&l [...]
+</ul>
+
+<p>You might have noticed the context objects of the <code>KeyedBroadcastProcessFunction</code>’s processing method. The context objects give access to additional functionality such as:</p>
+
+<ul>
+ <li>The broadcast state (read-write or read-only, depending on the method),</li>
+ <li>A <code>TimerService</code>, which gives access to the record’s timestamp, the current watermark, and which can register timers,</li>
+ <li>The current key (only available in <code>processElement()</code>), and</li>
+ <li>A method to apply a function the keyed state of each registered key (only available in <code>processBroadcastElement()</code>)</li>
+</ul>
+
+<p>The <code>KeyedBroadcastProcessFunction</code> has full access to Flink state and time features just like any other ProcessFunction and hence can be used to implement sophisticated application logic. Broadcast state was designed to be a versatile feature that adapts to different scenarios and use cases. Although we only discussed a fairly simple and restricted application, you can use broadcast state in many ways to implement the requirements of your application.</p>
+
+<h2 id="conclusion">Conclusion</h2>
+
+<p>In this blog post, we walked you through an example application to explain what Apache Flink’s broadcast state is and how it can be used to evaluate dynamic patterns on event streams. We’ve also discussed the API and showed the source code of our example application.</p>
+
+<p>We invite you to check the <a href="https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/broadcast_state.html">documentation</a> of this feature and provide feedback or suggestions for further improvements through our <a href="http://mail-archives.apache.org/mod_mbox/flink-community/">mailing list</a>.</p>
+</description>
+<pubDate>Wed, 26 Jun 2019 14:00:00 +0200</pubDate>
+<link>https://flink.apache.org/2019/06/26/broadcast-state.html</link>
+<guid isPermaLink="true">/2019/06/26/broadcast-state.html</guid>
+</item>
+
+<item>
+<title>A Deep-Dive into Flink's Network Stack</title>
+<description><style type="text/css">
+.tg {border-collapse:collapse;border-spacing:0;}
+.tg td{padding:10px 20px;border-style:solid;border-width:1px;overflow:hidden;word-break:normal;}
+.tg th{padding:10px 20px;border-style:solid;border-width:1px;overflow:hidden;word-break:normal;background-color:#eff0f1;}
+.tg .tg-wide{padding:10px 30px;}
+.tg .tg-top{vertical-align:top}
+.tg .tg-center{text-align:center;vertical-align:center}
+</style>
+
+<p>Flink’s network stack is one of the core components that make up the <code>flink-runtime</code> module and sit at the heart of every Flink job. It connects individual work units (subtasks) from all TaskManagers. This is where your streamed-in data flows through and it is therefore crucial to the performance of your Flink job for both the throughput as well as latency you observe. In contrast to the coordination channels between TaskManagers and JobManagers which are [...]
+
+<p>This blog post is the first in a series of posts about the network stack. In the sections below, we will first have a high-level look at what abstractions are exposed to the stream operators and then go into detail on the physical implementation and various optimisations Flink did. We will briefly present the result of these optimisations and Flink’s trade-off between throughput and latency. Future blog posts in this series will elaborate more on monitoring and metrics, tuning p [...]
+
+<div class="page-toc">
+<ul id="markdown-toc">
+ <li><a href="#logical-view" id="markdown-toc-logical-view">Logical View</a></li>
+ <li><a href="#physical-transport" id="markdown-toc-physical-transport">Physical Transport</a> <ul>
+ <li><a href="#inflicting-backpressure-1" id="markdown-toc-inflicting-backpressure-1">Inflicting Backpressure (1)</a></li>
+ </ul>
+ </li>
+ <li><a href="#credit-based-flow-control" id="markdown-toc-credit-based-flow-control">Credit-based Flow Control</a> <ul>
+ <li><a href="#inflicting-backpressure-2" id="markdown-toc-inflicting-backpressure-2">Inflicting Backpressure (2)</a></li>
+ <li><a href="#what-do-we-gain-where-is-the-catch" id="markdown-toc-what-do-we-gain-where-is-the-catch">What do we Gain? Where is the Catch?</a></li>
+ </ul>
+ </li>
+ <li><a href="#writing-records-into-network-buffers-and-reading-them-again" id="markdown-toc-writing-records-into-network-buffers-and-reading-them-again">Writing Records into Network Buffers and Reading them again</a> <ul>
+ <li><a href="#flushing-buffers-to-netty" id="markdown-toc-flushing-buffers-to-netty">Flushing Buffers to Netty</a></li>
+ <li><a href="#buffer-builder--buffer-consumer" id="markdown-toc-buffer-builder--buffer-consumer">Buffer Builder &amp; Buffer Consumer</a></li>
+ </ul>
+ </li>
+ <li><a href="#latency-vs-throughput" id="markdown-toc-latency-vs-throughput">Latency vs. Throughput</a></li>
+ <li><a href="#conclusion" id="markdown-toc-conclusion">Conclusion</a></li>
+</ul>
+
+</div>
+
+<h2 id="logical-view">Logical View</h2>
+
+<p>Flink’s network stack provides the following logical view to the subtasks when communicating with each other, for example during a network shuffle as required by a <code>keyBy()</code>.</p>
+
+<center>
+<img src="/img/blog/2019-06-05-network-stack/flink-network-stack1.png" width="400px" alt="Logical View on Flink's Network Stack" />
+</center>
+<p><br /></p>
+
+<p>It abstracts over the different settings of the following three concepts:</p>
+
+<ul>
+ <li>Subtask output type (<code>ResultPartitionType</code>):
+ <ul>
+ <li><strong>pipelined (bounded or unbounded):</strong>
+ Sending data downstream as soon as it is produced, potentially one-by-one, either as a bounded or unbounded stream of records.</li>
+ <li><strong>blocking:</strong>
+ Sending data downstream only when the full result was produced.</li>
+ </ul>
+ </li>
+ <li>Scheduling type:
+ <ul>
+ <li><strong>all at once (eager):</strong>
+ Deploy all subtasks of the job at the same time (for streaming applications).</li>
+ <li><strong>next stage on first output (lazy):</strong>
+ Deploy downstream tasks as soon as any of their producers generated output.</li>
+ <li><strong>next stage on complete output:</strong>
+ Deploy downstream tasks when any or all of their producers have generated their full output set.</li>
+ </ul>
+ </li>
+ <li>Transport:
+ <ul>
+ <li><strong>high throughput:</strong>
+ Instead of sending each record one-by-one, Flink buffers a bunch of records into its network buffers and sends them altogether. This reduces the costs per record and leads to higher throughput.</li>
+ <li><strong>low latency via buffer timeout:</strong>
+ By reducing the timeout of sending an incompletely filled buffer, you may sacrifice throughput for latency.</li>
+ </ul>
+ </li>
+</ul>
+
+<p>We will have a look at the throughput and low latency optimisations in the sections below which look at the physical layers of the network stack. For this part, let us elaborate a bit more on the output and scheduling types. First of all, it is important to know that the subtask output type and the scheduling type are closely intertwined making only specific combinations of the two valid.</p>
+
+<p>Pipelined result partitions are streaming-style outputs which need a live target subtask to send data to. The target can be scheduled before results are produced or at first output. Batch jobs produce bounded result partitions while streaming jobs produce unbounded results.</p>
+
+<p>Batch jobs may also produce results in a blocking fashion, depending on the operator and connection pattern that is used. In that case, the complete result must be produced first before the receiving task can be scheduled. This allows batch jobs to work more efficiently and with lower resource usage.</p>
+
+<p>The following table summarises the valid combinations:
+<br /></p>
+<center>
+<table class="tg">
+ <tr>
+ <th>Output Type</th>
+ <th>Scheduling Type</th>
+ <th>Applies to…</th>
+ </tr>
+ <tr>
+ <td rowspan="2">pipelined, unbounded</td>
+ <td>all at once</td>
+ <td>Streaming jobs</td>
+ </tr>
+ <tr>
+ <td>next stage on first output</td>
+ <td>n/a¹</td>
+ </tr>
+ <tr>
+ <td rowspan="2">pipelined, bounded</td>
+ <td>all at once</td>
+ <td>n/a²</td>
+ </tr>
+ <tr>
+ <td>next stage on first output</td>
+ <td>Batch jobs</td>
+ </tr>
+ <tr>
+ <td>blocking</td>
+ <td>next stage on complete output</td>
+ <td>Batch jobs</td>
+ </tr>
+</table>
+</center>
+<p><br /></p>
+
+<p><sup>1</sup> Currently not used by Flink. <br />
+<sup>2</sup> This may become applicable to streaming jobs once the <a href="/roadmap.html#batch-and-streaming-unification">Batch/Streaming unification</a> is done.</p>
+
+<p><br />
+Additionally, for subtasks with more than one input, scheduling start in two ways: after <em>all</em> or after <em>any</em> input producers to have produced a record/their complete dataset. For tuning the output types and scheduling decisions in batch jobs, please have a look at <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/api/java/org/apache/flink/api/common/ExecutionConfig.html#setExecutionMode-org.apache.flink.api.common.Executio [...]
+
+<p><br /></p>
+
+<h2 id="physical-transport">Physical Transport</h2>
+
+<p>In order to understand the physical data connections, please recall that, in Flink, different tasks may share the same slot via <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev/stream/operators/#task-chaining-and-resource-groups">slot sharing groups</a>. TaskManagers may also provide more than one slot to allow multiple subtasks of the same task to be scheduled onto the same TaskManager.</p>
+
+<p>For the example pictured below, we will assume a parallelism of 4 and a deployment with two task managers offering 2 slots each. TaskManager 1 executes subtasks A.1, A.2, B.1, and B.2 and TaskManager 2 executes subtasks A.3, A.4, B.3, and B.4. In a shuffle-type connection between task A and task B, for example from a <code>keyBy()</code>, there are 2x4 logical connections to handle on each TaskManager, some of which are local, some remote:
+<br /></p>
+
+<center>
+<table class="tg">
+ <tr>
+ <th></th>
+ <th class="tg-wide">B.1</th>
+ <th class="tg-wide">B.2</th>
+ <th class="tg-wide">B.3</th>
+ <th class="tg-wide">B.4</th>
+ </tr>
+ <tr>
+ <th class="tg-wide">A.1</th>
+ <td class="tg-center" colspan="2" rowspan="2">local</td>
+ <td class="tg-center" colspan="2" rowspan="2">remote</td>
+ </tr>
+ <tr>
+ <th class="tg-wide">A.2</th>
+ </tr>
+ <tr>
+ <th class="tg-wide">A.3</th>
+ <td class="tg-center" colspan="2" rowspan="2">remote</td>
+ <td class="tg-center" colspan="2" rowspan="2">local</td>
+ </tr>
+ <tr>
+ <th class="tg-wide">A.4</th>
+ </tr>
+</table>
+</center>
+
+<p><br /></p>
+
+<p>Each (remote) network connection between different tasks will get its own TCP channel in Flink’s network stack. However, if different subtasks of the same task are scheduled onto the same TaskManager, their network connections towards the same TaskManagers will be multiplexed and share a single TCP channel for reduced resource usage. In our example, this would apply to A.1 → B.3, A.1 → B.4, as well as A.2 → B.3, and A.2 → B.4 as pictured below:
+<br /></p>
+
+<center>
+<img src="/img/blog/2019-06-05-network-stack/flink-network-stack2.png" width="700px" alt="Physical-transport-Flink's Network Stack" />
+</center>
+<p><br /></p>
+
+<p>The results of each subtask are called <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/api/java/org/apache/flink/runtime/io/network/partition/ResultPartition.html">ResultPartition</a>, each split into separate <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/api/java/org/apache/flink/runtime/io/network/partition/ResultSubpartition.html">ResultSubpartitions</a> — one for each logical channel. At [...]
+
+<div class="highlight"><pre><code>#channels * buffers-per-channel + floating-buffers-per-gate
+</code></pre></div>
+
+<p>The total number of buffers on a single TaskManager usually does not need configuration. See the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/ops/config.html#configuring-the-network-buffers">Configuring the Network Buffers</a> documentation for details on how to do so if needed.</p>
+
+<h3 id="inflicting-backpressure-1">Inflicting Backpressure (1)</h3>
+
+<p>Whenever a subtask’s sending buffer pool is exhausted — buffers reside in either a result subpartition’s buffer queue or inside the lower, Netty-backed network stack — the producer is blocked, cannot continue, and experiences backpressure. The receiver works in a similar fashion: any incoming Netty buffer in the lower network stack needs to be made available to Flink via a network buffer. If there is no network buffer available in the appropriate subtask’s buffer pool, Flink wil [...]
+
+<p><br /></p>
+
+<center>
+<img src="/img/blog/2019-06-05-network-stack/flink-network-stack3.png" width="700px" alt="Physical-transport-backpressure-Flink's Network Stack" />
+</center>
+<p><br /></p>
+
+<p>To prevent this situation from even happening, Flink 1.5 introduced its own flow control mechanism.</p>
+
+<p><br /></p>
+
+<h2 id="credit-based-flow-control">Credit-based Flow Control</h2>
+
+<p>Credit-based flow control makes sure that whatever is “on the wire” will have capacity at the receiver to handle. It is based on the availability of network buffers as a natural extension of the mechanisms Flink had before. Instead of only having a shared local buffer pool, each remote input channel now has its own set of <strong>exclusive buffers</strong>. Conversely, buffers in the local buffer pool are called <strong>floating buffers</strong> as they w [...]
+
+<p>Receivers will announce the availability of buffers as <strong>credits</strong> to the sender (1 buffer = 1 credit). Each result subpartition will keep track of its <strong>channel credits</strong>. Buffers are only forwarded to the lower network stack if credit is available and each sent buffer reduces the credit score by one. In addition to the buffers, we also send information about the current <strong>backlog</strong> size which specifies [...]
+<br /></p>
+
+<center>
+<img src="/img/blog/2019-06-05-network-stack/flink-network-stack4.png" width="700px" alt="Physical-transport-credit-flow-Flink's Network Stack" />
+</center>
+<p><br /></p>
+
+<p>Credit-based flow control will use <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/ops/config.html#taskmanager-network-memory-buffers-per-channel">buffers-per-channel</a> to specify how many buffers are exclusive (mandatory) and <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/ops/config.html#taskmanager-network-memory-floating-buffers-per-gate">floating-buffers-per-gate</a> for the local buffer [...]
+<br /></p>
+
+<p><sup>3</sup>If there are not enough buffers available, each buffer pool will get the same share of the globally available ones (± 1).</p>
+
+<h3 id="inflicting-backpressure-2">Inflicting Backpressure (2)</h3>
+
+<p>As opposed to the receiver’s backpressure mechanisms without flow control, credits provide a more direct control: If a receiver cannot keep up, its available credits will eventually hit 0 and stop the sender from forwarding buffers to the lower network stack. There is backpressure on this logical channel only and there is no need to block reading from a multiplexed TCP channel. Other receivers are therefore not affected in processing available buffers.</p>
+
+<h3 id="what-do-we-gain-where-is-the-catch">What do we Gain? Where is the Catch?</h3>
+
+<p><img align="right" src="/img/blog/2019-06-05-network-stack/flink-network-stack5.png" width="300" height="200" alt="Physical-transport-credit-flow-checkpoints-Flink's Network Stack" /></p>
+
+<p>Since, with flow control, a channel in a multiplex cannot block another of its logical channels, the overall resource utilisation should increase. In addition, by having full control over how much data is “on the wire”, we are also able to improve <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/internals/stream_checkpointing.html#checkpointing">checkpoint alignments</a>: without flow control, it would take a while for the channel to fil [...]
+
+<p>However, the additional announce messages from the receiver may come at some additional costs, especially in setup using SSL-encrypted channels. Also, a single input channel cannot make use of all buffers in the buffer pool because exclusive buffers are not shared. It can also not start right away with sending as much data as is available so that during ramp-up (if you are producing data faster than announcing credits in return) it may take longer to send data through. While thi [...]
+
+<p>There is one more thing you may notice when using credit-based flow control: since we buffer less data between the sender and receiver, you may experience backpressure earlier. This is, however, desired and you do not really get any advantage by buffering more data. If you want to buffer more but keep flow control, you could consider increasing the number of floating buffers via <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/ops/config.html#taskmana [...]
+
+<p><br /></p>
+
+<center>
+<table class="tg">
+ <tr>
+ <th>Advantages</th>
+ <th>Disadvantages</th>
+ </tr>
+ <tr>
+ <td class="tg-top">
+ • better resource utilisation with data skew in multiplexed connections <br /><br />
+ • improved checkpoint alignment<br /><br />
+ • reduced memory use (less data in lower network layers)</td>
+ <td class="tg-top">
+ • additional credit-announce messages<br /><br />
+ • additional backlog-announce messages (piggy-backed with buffer messages, almost no overhead)<br /><br />
+ • potential round-trip latency</td>
+ </tr>
+ <tr>
+ <td class="tg-center" colspan="2">• backpressure appears earlier</td>
+ </tr>
+</table>
+</center>
+<p><br /></p>
+
+<div class="alert alert-info">
+ <p><span class="label label-info" style="display: inline-block"><span class="glyphicon glyphicon-info-sign" aria-hidden="true"></span> Note</span>
+If you need to turn off credit-based flow control, you can add this to your <code>flink-conf.yaml</code>:</p>
+
+ <p><code>taskmanager.network.credit-model: false</code></p>
+
+ <p>This parameter, however, is deprecated and will eventually be removed along with the non-credit-based flow control code.</p>
+</div>
+
+<p><br /></p>
+
+<h2 id="writing-records-into-network-buffers-and-reading-them-again">Writing Records into Network Buffers and Reading them again</h2>
+
+<p>The following picture extends the slightly more high-level view from above with further details of the network stack and its surrounding components, from the collection of a record in your sending operator to the receiving operator getting it:
+<br /></p>
+
+<center>
+<img src="/img/blog/2019-06-05-network-stack/flink-network-stack6.png" width="700px" alt="Physical-transport-complete-Flink's Network Stack" />
+</center>
+<p><br /></p>
+
+<p>After creating a record and passing it along, for example via <code>Collector#collect()</code>, it is given to the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/api/java/org/apache/flink/runtime/io/network/api/writer/RecordWriter.html">RecordWriter</a> which serialises the record from a Java object into a sequence of bytes which eventually ends up in a network buffer that is handed along as described above. The RecordWrite [...]
+
+<p>On the receiver’s side, the lower network stack (netty) is writing received buffers into the appropriate input channels. The (stream) tasks’s thread eventually reads from these queues and tries to deserialise the accumulated bytes into Java objects with the help of the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/api/java/org/apache/flink/runtime/io/network/api/reader/RecordReader.html">RecordReader</a> and going through the <a hr [...]
+<br /></p>
+
+<h3 id="flushing-buffers-to-netty">Flushing Buffers to Netty</h3>
+
+<p>In the picture above, the credit-based flow control mechanics actually sit inside the “Netty Server” (and “Netty Client”) components and the buffer the RecordWriter is writing to is always added to the result subpartition in an empty state and then gradually filled with (serialised) records. But when does Netty actually get the buffer? Obviously, it cannot take bytes whenever they become available since that would not only add substantial costs due to cross-thread communication [...]
+
+<p>In Flink, there are three situations that make a buffer available for consumption by the Netty server:</p>
+
+<ul>
+ <li>a buffer becomes full when writing a record to it, or<br /></li>
+ <li>the buffer timeout hits, or<br /></li>
+ <li>a special event such as a checkpoint barrier is sent.<br />
+<br /></li>
+</ul>
+
+<h4 id="flush-after-buffer-full">Flush after Buffer Full</h4>
+
+<p>The RecordWriter works with a local serialisation buffer for the current record and will gradually write these bytes to one or more network buffers sitting at the appropriate result subpartition queue. Although a RecordWriter can work on multiple subpartitions, each subpartition has only one RecordWriter writing data to it. The Netty server, on the other hand, is reading from multiple result subpartitions and multiplexing the appropriate ones into a single channel as described a [...]
+<br /></p>
+
+<center>
+<img src="/img/blog/2019-06-05-network-stack/flink-network-stack7.png" width="500px" alt="Record-writer-to-network-Flink's Network Stack" />
+</center>
+<p><br /></p>
+
+<p><sup>4</sup>We can assume it already got the notification if there are more finished buffers in the queue.
+<br /></p>
+
+<h4 id="flush-after-buffer-timeout">Flush after Buffer Timeout</h4>
+
+<p>In order to support low-latency use cases, we cannot only rely on buffers being full in order to send data downstream. There may be cases where a certain communication channel does not have too many records flowing through and unnecessarily increase the latency of the few records you actually have. Therefore, a periodic process will flush whatever data is available down the stack: the output flusher. The periodic interval can be configured via <a href="https://ci.apache. [...]
+<br /></p>
+
+<center>
+<img src="/img/blog/2019-06-05-network-stack/flink-network-stack8.png" width="500px" alt="Record-writer-to-network-with-flusher-Flink's Network Stack" />
+</center>
+<p><br /></p>
+
+<p><sup>5</sup>Strictly speaking, the output flusher does not give any guarantees - it only sends a notification to Netty which can pick it up at will / capacity. This also means that the output flusher has no effect if the channel is backpressured.
+<br /></p>
+
+<h4 id="flush-after-special-event">Flush after special event</h4>
+
+<p>Some special events also trigger immediate flushes if being sent through the RecordWriter. The most important ones are checkpoint barriers or end-of-partition events which obviously should go quickly and not wait for the output flusher to kick in.
+<br /></p>
+
+<h4 id="further-remarks">Further remarks</h4>
+
+<p>In contrast to Flink &lt; 1.5, please note that (a) network buffers are now placed in the subpartition queues directly and (b) we are not closing the buffer on each flush. This gives us a few advantages:</p>
+
+<ul>
+ <li>less synchronisation overhead (output flusher and RecordWriter are independent)</li>
+ <li>in high-load scenarios where Netty is the bottleneck (either through backpressure or directly), we can still accumulate data in incomplete buffers</li>
+ <li>significant reduction of Netty notifications</li>
+</ul>
+
+<p>However, you may notice an increased CPU use and TCP packet rate during low load scenarios. This is because, with the changes, Flink will use any <em>available</em> CPU cycles to try to maintain the desired latency. Once the load increases, this will self-adjust by buffers filling up more. High load scenarios are not affected and even get a better throughput because of the reduced synchronisation overhead.
+<br /></p>
+
+<h3 id="buffer-builder--buffer-consumer">Buffer Builder &amp; Buffer Consumer</h3>
+
+<p>If you want to dig deeper into how the producer-consumer mechanics are implemented in Flink, please take a closer look at the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/api/java/org/apache/flink/runtime/io/network/buffer/BufferBuilder.html">BufferBuilder</a> and <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/api/java/org/apache/flink/runtime/io/network/buffer/BufferConsumer.html">BufferConsumer [...]
+
+<p><br /></p>
+
+<h2 id="latency-vs-throughput">Latency vs. Throughput</h2>
+
+<p>Network buffers were introduced to get higher resource utilisation and higher throughput at the cost of having some records wait in buffers a little longer. Although an upper limit to this wait time can be given via the buffer timeout, you may be curious to find out more about the trade-off between these two dimensions: latency and throughput, as, obviously, you cannot get both. The following plot shows various values for the buffer timeout starting at 0 (flush with every record [...]
+<br /></p>
+
+<center>
+<img src="/img/blog/2019-06-05-network-stack/flink-network-stack9.png" width="650px" alt="Network-buffertimeout-Flink's Network Stack" />
+</center>
+<p><br /></p>
+
+<p>As you can see, with Flink 1.5+, even very low buffer timeouts such as 1ms (for low-latency scenarios) provide a maximum throughput as high as 75% of the default timeout where more data is buffered before being sent over the wire.</p>
+
+<p><br /></p>
+
+<h2 id="conclusion">Conclusion</h2>
+
+<p>Now you know about result partitions, the different network connections and scheduling types for both batch and streaming. You also know about credit-based flow control and how the network stack works internally, in order to reason about network-related tuning parameters and about certain job behaviours. Future blog posts in this series will build upon this knowledge and go into more operational details including relevant metrics to look at, further network stack tuning, and com [...]
+
+</description>
+<pubDate>Wed, 05 Jun 2019 10:45:00 +0200</pubDate>
+<link>https://flink.apache.org/2019/06/05/flink-network-stack.html</link>
+<guid isPermaLink="true">/2019/06/05/flink-network-stack.html</guid>
+</item>
+
+<item>
+<title>State TTL in Flink 1.8.0: How to Automatically Cleanup Application State in Apache Flink</title>
+<description><p>A common requirement for many stateful streaming applications is to automatically cleanup application state for effective management of your state size, or to control how long the application state can be accessed (e.g. due to legal regulations like the GDPR). The state time-to-live (TTL) feature was initiated in Flink 1.6.0 and enabled application state cleanup and efficient state size management in Apache Flink.</p>
+
+<p>In this post, we motivate the State TTL feature and discuss its use cases. Moreover, we show how to use and configure it. We explain how Flink internally manages state with TTL and present some exciting additions to the feature in Flink 1.8.0. The blog post concludes with an outlook on future improvements and extensions.</p>
+
+<h1 id="the-transient-nature-of-state">The Transient Nature of State</h1>
+
+<p>There are two major reasons why state should be maintained only for a limited time. For example, let’s imagine a Flink application that ingests a stream of user login events and stores for each user the time of the last login to improve the experience of frequent visitors.</p>
+
+<ul>
+ <li>
+ <p><strong>Controlling the size of state.</strong>
+Being able to efficiently manage an ever-growing state size is a primary use case for state TTL. Oftentimes, data needs to be persisted temporarily while there is some user activity around it, e.g. web sessions. When the activity ends there is no longer interest in that data while it still occupies storage. Flink 1.8.0 introduces background cleanup of old state based on TTL that makes the eviction of no-longer-necessary data frictionless. Previously, the application developer had to take [...]
+ </li>
+ <li>
+ <p><strong>Complying with data protection and sensitive data requirements.</strong>
+Recent developments around data privacy regulations, such as the General Data Protection Regulation (GDPR) introduced by the European Union, make compliance with such data requirements or treating sensitive data a top priority for many use cases and applications. An example of such use cases includes applications that require keeping data for a specific timeframe and preventing access to it thereafter. This is a common challenge for companies providing short-term services to their custom [...]
+ </li>
+</ul>
+
+<p>Both requirements can be addressed by a feature that periodically, yet continuously, removes the state for a key once it becomes unnecessary or unimportant and there is no requirement to keep it in storage any more.</p>
+
+<h1 id="state-ttl-for-continuous-cleanup-of-application-state">State TTL for continuous cleanup of application state</h1>
+
+<p>The 1.6.0 release of Apache Flink introduced the State TTL feature. It enabled developers of stream processing applications to configure the state of operators to expire and be cleaned up after a defined timeout (time-to-live). In Flink 1.8.0 the feature was extended, including continuous cleanup of old entries for both the RocksDB and the heap state backends (FSStateBackend and MemoryStateBackend), enabling a continuous cleanup process of old entries (according to the TTL setti [...]
+
+<p>In Flink’s DataStream API, application state is defined by a <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev/stream/state/state.html#using-managed-keyed-state">state descriptor</a>. State TTL is configured by passing a <code>StateTtlConfiguration</code> object to a state descriptor. The following Java example shows how to create a state TTL configuration and provide it to the state descriptor that holds the last login ti [...]
+
+<div class="highlight"><pre><code class="language-java"><span class="kn">import</span> <span class="nn">org.apache.flink.api.common.state.StateTtlConfig</span><span class="o">;</span>
+<span class="kn">import</span> <span class="nn">org.apache.flink.api.common.time.Time</span><span class="o">;</span>
+<span class="kn">import</span> <span class="nn">org.apache.flink.api.common.state.ValueStateDescriptor</span><span class="o">;</span>
+
+<span class="n">StateTtlConfig</span> <span class="n">ttlConfig</span> <span class="o">=</span> <span class="n">StateTtlConfig</span>
+ <span class="o">.</span><span class="na">newBuilder</span><span class="o">(</span><span class="n">Time</span><span class="o">.</span><span class="na">days</span><span class="o">(</span><span class="mi">7</span><span class="o">))</span>
+ <span class="o">.</span><span class="na">setUpdateType</span><span class="o">(</span><span class="n">StateTtlConfig</span><span class="o">.</span><span class="na">UpdateType</span><span class="o">.</span><span class="na">OnCreateAndWrite</span><span class="o">)</span>
+ <span class="o">.</span><span class="na">setStateVisibility</span><span class="o">(</span><span class="n">StateTtlConfig</span><span class="o">.</span><span class="na">StateVisibility</span><span class="o">.</span><span class="na">NeverReturnExpired</span><span class="o">)</span>
+ <span class="o">.</span><span class="na">build</span><span class="o">();</span>
+
+<span class="n">ValueStateDescriptor</span><span class="o">&lt;</span><span class="n">Long</span><span class="o">&gt;</span> <span class="n">lastUserLogin</span> <span class="o">=</span>
+ <span class="k">new</span> <span class="n">ValueStateDescriptor</span><span class="o">&lt;&gt;(</span><span class="s">&quot;lastUserLogin&quot;</span><span class="o">,</span> <span class="n">Long</span><span class="o">.</span><span class="na">class</span><span class="o">);</span>
+
+<span class="n">lastUserLogin</span><span class="o">.</span><span class="na">enableTimeToLive</span><span class="o">(</span><span class="n">ttlConfig</span><span class="o">);</span></code></pre></div>
+
+<p>Flink provides multiple options to configure the behavior of the state TTL functionality.</p>
+
+<ul>
+ <li>
+ <p><strong>When is the Time-to-Live reset?</strong>
+By default, the expiration time of a state entry is updated when the state is modified. Optionally, it can also be updated on read access at the cost of an additional write operation to update the timestamp.</p>
+ </li>
+ <li>
+ <p><strong>Can the expired state be accessed one last time?</strong>
+State TTL employs a lazy strategy to clean up expired state. This can lead to the situation that an application attempts to read state which is expired but hasn’t been removed yet. You can configure whether such a read request returns the expired state or not. In either case, the expired state is immediately removed afterwards. While the option of returning expired state favors data availability, not returning expired state can be required for data protection regulations.</p>
+ </li>
+ <li>
+ <p><strong>Which time semantics are used for the Time-to-Live timers?</strong>
+With Flink 1.8.0, users can only define a state TTL in terms of processing time. The support for event time is planned for future Apache Flink releases.</p>
+ </li>
+</ul>
+
+<p>You can read more about how to use state TTL in the <a href="https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/state.html#state-time-to-live-ttl">Apache Flink documentation</a>.</p>
+
+<p>Internally, the State TTL feature is implemented by storing an additional timestamp of the last relevant state access, along with the actual state value. While this approach adds some storage overhead, it allows Flink to check for the expired state during state access, checkpointing, recovery, or dedicated storage cleanup procedures.</p>
+
+<h1 id="taking-out-the-garbage">“Taking out the Garbage”</h1>
+
+<p>When a state object is accessed in a read operation, Flink will check its timestamp and clear the state if it is expired (depending on the configured state visibility, the expired state is returned or not). Due to this lazy removal, expired state that is never accessed again will forever occupy storage space unless it is garbage collected.</p>
+
+<p>So how can the expired state be removed without the application logic explicitly taking care of it? In general, there are different possible strategies to remove it in the background.</p>
+
+<h2 id="keep-full-state-snapshots-clean">Keep full state snapshots clean</h2>
+
+<p>Flink 1.6.0 already supported automatic eviction of the expired state when a full snapshot for a checkpoint or savepoint is taken. Note that state eviction is not applied for incremental checkpoints. State eviction on full snapshots must be explicitly enabled as shown in the following example:</p>
+
+<div class="highlight"><pre><code class="language-java"><span class="n">StateTtlConfig</span> <span class="n">ttlConfig</span> <span class="o">=</span> <span class="n">StateTtlConfig</span>
+ <span class="o">.</span><span class="na">newBuilder</span><span class="o">(</span><span class="n">Time</span><span class="o">.</span><span class="na">days</span><span class="o">(</span><span class="mi">7</span><span class="o">))</span>
+ <span class="o">.</span><span class="na">cleanupFullSnapshot</span><span class="o">()</span>
+ <span class="o">.</span><span class="na">build</span><span class="o">();</span></code></pre></div>
+
+<p>The local storage stays untouched but the size of the stored snapshot is reduced. The local state of an operator will only be cleaned up when the operator reloads its state from a snapshot, i.e. in case of recovery or when starting from a savepoint.</p>
+
+<p>Due to these limitations, applications still need to actively remove state after it expired in Flink 1.6.0. To improve the user experience, Flink 1.8.0 introduces two more autonomous cleanup strategies, one for each of Flink’s two state backend types. We describe them below.</p>
+
+<h2 id="incremental-cleanup-in-heap-state-backends">Incremental cleanup in Heap state backends</h2>
+
+<p>This approach is specific to the Heap state backends (FSStateBackend and MemoryStateBackend). The idea is that the storage backend keeps a lazy global iterator over all state entries. Certain events, for instance state access, trigger an incremental cleanup. Every time an incremental cleanup is triggered, the iterator is advanced. The traversed state entries are checked and expired once are removed. The following code example shows how to enable incremental cleanup:</p>
+
+<div class="highlight"><pre><code class="language-java"><span class="n">StateTtlConfig</span> <span class="n">ttlConfig</span> <span class="o">=</span> <span class="n">StateTtlConfig</span>
+ <span class="o">.</span><span class="na">newBuilder</span><span class="o">(</span><span class="n">Time</span><span class="o">.</span><span class="na">days</span><span class="o">(</span><span class="mi">7</span><span class="o">))</span>
+ <span class="c1">// check 10 keys for every state access</span>
+ <span class="o">.</span><span class="na">cleanupIncrementally</span><span class="o">(</span><span class="mi">10</span><span class="o">,</span> <span class="kc">false</span><span class="o">)</span>
+ <span class="o">.</span><span class="na">build</span><span class="o">();</span></code></pre></div>
+
+<p>If enabled, every state access triggers a cleanup step. For every clean up step, a certain number of state entries are checked for expiration. There are two tuning parameters. The first defines the number of state entries to check for each cleanup step. The second parameter is a flag to trigger a cleanup step after each processed record, additionally to each state access.</p>
+
+<p>There are two important caveats about this approach:
+* The first one is that the time spent for the incremental cleanup increases the record processing latency.
+* The second one should be practically negligible but still worth mentioning: if no state is accessed or no records are processed, expired state won’t be removed.</p>
+
+<h2 id="rocksdb-background-compaction-to-filter-out-expired-state">RocksDB background compaction to filter out expired state</h2>
+
+<p>If your application uses the RocksDB state backend, you can enable another cleanup strategy which is based on a Flink specific compaction filter. RocksDB periodically runs asynchronous compactions to merge state updates and reduce storage. The Flink compaction filter checks the expiration timestamp of state entries with TTL and discards all expired values.</p>
+
+<p>The first step to activate this feature is to configure the RocksDB state backend by setting the following Flink configuration option: <code>state.backend.rocksdb.ttl.compaction.filter.enabled</code>. Once the RocksDB state backend is configured, the compaction cleanup strategy is enabled for a state as shown in the following code example:</p>
+
+<div class="highlight"><pre><code class="language-java"><span class="n">StateTtlConfig</span> <span class="n">ttlConfig</span> <span class="o">=</span> <span class="n">StateTtlConfig</span>
+ <span class="o">.</span><span class="na">newBuilder</span><span class="o">(</span><span class="n">Time</span><span class="o">.</span><span class="na">days</span><span class="o">(</span><span class="mi">7</span><span class="o">))</span>
+ <span class="o">.</span><span class="na">cleanupInRocksdbCompactFilter</span><span class="o">()</span>
+ <span class="o">.</span><span class="na">build</span><span class="o">();</span></code></pre></div>
+<p>Keep in mind that calling the Flink TTL filter slows down the RocksDB compaction.</p>
+
+<h2 id="eager-state-cleanup-with-timers">Eager State Cleanup with Timers</h2>
+
+<p>Another way to manually cleanup state is based on Flink timers. This is an idea that the community is currently evaluating for future releases. With this approach, a cleanup timer is registered for every state access. This approach is more predictable because state is eagerly removed as soon as it expires. However, it is more expensive because the timers consume storage along with the original state.</p>
+
+<h1 id="future-work">Future work</h1>
+
+<p>Apart from including the timer-based cleanup strategy, mentioned above, the Flink community has plans to further improve the state TTL feature. The possible improvements include adding support of TTL for event time scale (only processing time is supported at the moment) and enabling State TTL for queryable state.</p>
+
+<p>We encourage you to join the conversation and share your thoughts and ideas in the <a href="https://issues.apache.org/jira/projects/FLINK/summary">Apache Flink JIRA board</a> or by subscribing to the Apache Flink dev mailing list. Feedback or suggestions are always appreciated and we look forward to hearing your thoughts on the Flink mailing lists.</p>
+
+<h1 id="summary">Summary</h1>
+
+<p>Time-based state access restrictions and controlling the size of application state are common challenges in the world of stateful stream processing. Flink’s 1.8.0 release significantly improves the State TTL feature by adding support for continuous background cleanup of expired state objects. The new clean up mechanisms relieve you from manually implementing state cleanup. They are also more efficient due to their lazy nature. State TTL gives you control over the size of your ap [...]
+</description>
+<pubDate>Sun, 19 May 2019 14:00:00 +0200</pubDate>
+<link>https://flink.apache.org/2019/05/19/state-ttl.html</link>
+<guid isPermaLink="true">/2019/05/19/state-ttl.html</guid>
+</item>
+
+<item>
+<title>Flux capacitor, huh? Temporal Tables and Joins in Streaming SQL</title>
+<description><p>Figuring out how to manage and model temporal data for effective point-in-time analysis was a longstanding battle, dating as far back as the early 80’s, that culminated with the introduction of temporal tables in the SQL standard in 2011. Up to that point, users were doomed to implement this as part of the application logic, often hurting the length of the development lifecycle as well as the maintainability of the code. And, although there isn’t a single, commonly [...]
+
+<center>
+<img src="/img/blog/2019-05-13-temporal-tables/TemporalTables1.png" width="500px" alt="Taxi Fares and Conversion Rates" />
+</center>
+<p><br /></p>
+
+<p><strong>For example:</strong> given a stream with Taxi Fare events tied to the local currency of the ride location, we might want to convert the fare price to a common currency for further processing. As conversion rates excel at fluctuating over time, each Taxi Fare event would need to be matched to the rate that was valid at the time the event occurred in order to produce a reliable result.</p>
+
+<h2 id="modelling-temporal-data-with-flink">Modelling Temporal Data with Flink</h2>
+
+<p>In the 1.7 release, Flink has introduced the concept of <strong>temporal tables</strong> into its streaming SQL and Table API: parameterized views on append-only tables — or, any table that only allows records to be inserted, never updated or deleted — that are interpreted as a changelog and keep data closely tied to time context, so that it can be interpreted as valid only within a specific period of time. Transforming a stream into a temporal table requires:</p>
+
+<ul>
+ <li>
+ <p>Defining a <strong>primary key</strong> and a <strong>versioning field</strong> that can be used to keep track of the changes that happen over time;</p>
+ </li>
+ <li>
+ <p>Exposing the stream as a <strong>temporal table function</strong> that maps each point in time to a static relation.</p>
+ </li>
+</ul>
+
+<p>Going back to our example use case, a temporal table is just what we need to model the conversion rate data such as to make it useful for point-in-time querying. Temporal table functions are implemented as an extension of Flink’s generic <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev/table/udfs.html#table-functions">table function</a> class and can be defined in the same straightforward way to be used with the Table API or SQL pars [...]
+
+<div class="highlight"><pre><code class="language-java"><span class="kn">import</span> <span class="nn">org.apache.flink.table.functions.TemporalTableFunction</span><span class="o">;</span>
+
+<span class="o">(...)</span>
+
+<span class="c1">// Get the stream and table environments.</span>
+<span class="n">StreamExecutionEnvironment</span> <span class="n">env</span> <span class="o">=</span> <span class="n">StreamExecutionEnvironment</span><span class="o">.</span><span class="na">getExecutionEnvironment</span><span class="o">();</span>
+<span class="n">StreamTableEnvironment</span> <span class="n">tEnv</span> <span class="o">=</span> <span class="n">StreamTableEnvironment</span><span class="o">.</span><span class="na">getTableEnvironment</span><span class="o">(</span><span class="n">env</span><span class="o">);</span>
+
+<span class="c1">// Provide a sample static data set of the rates history table.</span>
+<span class="n">List</span> <span class="o">&lt;</span><span class="n">Tuple2</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">Long</span><span class="o">&gt;&gt;</span><span class="n">ratesHistoryData</span> <span class= [...]
+
+<span class="n">ratesHistoryData</span><span class="o">.</span><span class="na">add</span><span class="o">(</span><span class="n">Tuple2</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="s">&quot;USD&quot;</span><span class="o"> [...]
+<span class="n">ratesHistoryData</span><span class="o">.</span><span class="na">add</span><span class="o">(</span><span class="n">Tuple2</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="s">&quot;EUR&quot;</span><span class="o"> [...]
+<span class="n">ratesHistoryData</span><span class="o">.</span><span class="na">add</span><span class="o">(</span><span class="n">Tuple2</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="s">&quot;YEN&quot;</span><span class="o"> [...]
+<span class="n">ratesHistoryData</span><span class="o">.</span><span class="na">add</span><span class="o">(</span><span class="n">Tuple2</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="s">&quot;EUR&quot;</span><span class="o"> [...]
+<span class="n">ratesHistoryData</span><span class="o">.</span><span class="na">add</span><span class="o">(</span><span class="n">Tuple2</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="s">&quot;USD&quot;</span><span class="o"> [...]
+
+<span class="c1">// Create and register an example table using the sample data set.</span>
+<span class="n">DataStream</span><span class="o">&lt;</span><span class="n">Tuple2</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">Long</span><span class="o">&gt;&gt;</span> <span class="n">ratesHistoryStream</span> <spa [...]
+
+<span class="n">Table</span> <span class="n">ratesHistory</span> <span class="o">=</span> <span class="n">tEnv</span><span class="o">.</span><span class="na">fromDataStream</span><span class="o">(</span><span class="n">ratesHistoryStream</span><span class="o">,</span> <span class="s&qu [...]
+
+<span class="n">tEnv</span><span class="o">.</span><span class="na">registerTable</span><span class="o">(</span><span class="s">&quot;RatesHistory&quot;</span><span class="o">,</span> <span class="n">ratesHistory</span><span class="o">);</span>
+
+<span class="c1">// Create and register the temporal table function &quot;rates&quot;.</span>
+<span class="c1">// Define &quot;r_proctime&quot; as the versioning field and &quot;r_currency&quot; as the primary key.</span>
+<span class="n">TemporalTableFunction</span> <span class="n">rates</span> <span class="o">=</span> <span class="n">ratesHistory</span><span class="o">.</span><span class="na">createTemporalTableFunction</span><span class="o">(</span><span class="s">&quot;r_proctime&quot;</span><span class="o"&g [...]
+
+<span class="n">tEnv</span><span class="o">.</span><span class="na">registerFunction</span><span class="o">(</span><span class="s">&quot;Rates&quot;</span><span class="o">,</span> <span class="n">rates</span><span class="o">);</span>
+
+<span class="o">(...)</span></code></pre></div>
+
+<p>What does this <strong>Rates</strong> function do, in practice? Imagine we would like to check what the conversion rates looked like at a given time — say, 11:00. We could simply do something like:</p>
+
+<div class="highlight"><pre><code class="language-sql"><span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">Rates</span><span class="p">(</span><span class="s1">&#39;11:00&#39;</span><span class="p">);</span></code></pre></div>
+
+<center>
+<img src="/img/blog/2019-05-13-temporal-tables/TemporalTables2.png" width="650px" alt="Point-in-time Querying" />
+</center>
+<p><br /></p>
+
+<p>Even though Flink does not yet support querying temporal table functions with a constant time attribute parameter, these functions can be used to cover a much more interesting scenario: temporal table joins.</p>
+
+<h2 id="streaming-joins-using-temporal-tables">Streaming Joins using Temporal Tables</h2>
+
+<p>Temporal tables reach their full potential when used in combination — erm, joined — with streaming data, for instance to power applications that must continuously whitelist against a reference dataset that changes over time for auditing or regulatory compliance. While efficient joins have long been an enduring challenge for query processors due to computational cost and resource consumption, joins over streaming data carry some additional challenges:</p>
+
+<ul>
+ <li>The <strong>unbounded</strong> nature of streams means that inputs are continuously evaluated and intermediate join results can consume memory resources indefinitely. Flink gracefully manages its memory consumption out-of-the-box (even for heavier cases where joins require spilling to disk) and supports time-windowed joins to bound the amount of data that needs to be kept around as state;</li>
+ <li>Streaming data might be <strong>out-of-order</strong> and <strong>late</strong>, so it is not possible to enforce an ordering upfront and time handling requires some thinking to avoid unnecessary outputs and retractions.</li>
+</ul>
+
+<p>In the particular case of temporal data, time-windowed joins are not enough (well, at least not without getting into some expensive tweaking): sooner or later, each reference record will fall outside of the window and be wiped from state, no longer being considered for future join results. To address this limitation, Flink has introduced support for temporal table joins to cover time-varying relations.</p>
+
+<center>
+<img src="/img/blog/2019-05-13-temporal-tables/TemporalTables3.png" width="500px" alt="Temporal Table Join between Taxi Fares and Conversion Rates" />
+</center>
+<p><br /></p>
+
+<p>Each record from the append-only table on the probe side (<code>Taxi Fare</code>) is joined with the version of the record from the temporal table on the build side (<code>Conversion Rate</code>) that most closely matches the probe side record time attribute (<code>time</code>) for the same value of the primary key (<code>currency</code>). Remember the temporal table function (<code>Rates</code>) we registered earlier? [...]
+
+<center>
+<img src="/img/blog/2019-05-13-temporal-tables/TemporalTables4.png" width="700px" alt="Regular Join vs. Temporal Table Join" />
+</center>
+<p><br /></p>
+
+<p>Temporal table joins support both <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev/table/streaming/joins.html#processing-time-temporal-joins">processing</a> and <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev/table/streaming/joins.html#event-time-temporal-joins">event time</a> semantics and effectively limit the amount of data kept in state while also allowing records on the build side t [...]
+
+<ul>
+ <li>Narrowing the <strong>scope</strong> of the join: only the time-matching version of <code>ratesHistory</code> is visible for a given <code>taxiFare.time</code>;</li>
+ <li>Pruning <strong>unneeded records</strong> from state: for cases using event time, records between current time and the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev/event_time.html#event-time-and-watermarks">watermark</a> delay are persisted for both the probe and build side. These are discarded as soon as the watermark arrives and the results are emitted — allowing the join operation to move forward in time and the [...]
+</ul>
+
+<h2 id="conclusion">Conclusion</h2>
+
+<p>All this means it is now possible to express continuous stream enrichment in relational and time-varying terms using Flink without dabbling into syntactic patchwork or compromising performance. In other words: stream time-travelling minus the flux capacitor. Extending this syntax to batch processing for enriching historic data with proper (event) time semantics is also part of the Flink roadmap!</p>
+
+<p>If you’d like to get some <strong>hands-on practice in joining streams with Flink SQL</strong> (and Flink SQL in general), checkout this <a href="https://github.com/ververica/sql-training/wiki">free training for Flink SQL</a>. The training environment is based on Docker and set up in just a few minutes.</p>
+
+<p>Subscribe to the <a href="/community.html#mailing-lists">Apache Flink mailing lists</a> to stay up-to-date with the latest developments in this space.</p>
+</description>
+<pubDate>Tue, 14 May 2019 14:00:00 +0200</pubDate>
+<link>https://flink.apache.org/2019/05/14/temporal-tables.html</link>
+<guid isPermaLink="true">/2019/05/14/temporal-tables.html</guid>
+</item>
+
+<item>
+<title>When Flink & Pulsar Come Together</title>
+<description><p>The open source data technology frameworks <a href="https://flink.apache.org/">Apache Flink</a> and <a href="https://pulsar.apache.org/en/">Apache Pulsar</a> can integrate in different ways to provide elastic data processing at large scale. I recently gave a talk at <a href="https://www.flink-forward.org/">Flink Forward</a> San Francisco 2019 and presented some of the integrations between the two fram [...]
+
+<h2 id="a-brief-introduction-to-apache-pulsar">A brief introduction to Apache Pulsar</h2>
+
+<p><a href="https://pulsar.apache.org/en/">Apache Pulsar</a> is an open-source distributed pub-sub messaging system under the stewardship of the <a href="https://www.apache.org/">Apache Software Foundation</a>. Pulsar is a multi-tenant, high-performance solution for server-to-server messaging including multiple features such as native support for multiple clusters in a Pulsar instance, with seamless <a href="https://pulsar.apache. [...]
+
+<p>The first differentiating factor stems from the fact that although Pulsar provides a flexible pub-sub messaging system it is also backed by durable log storage — hence combining both messaging and storage under one framework. Because of that layered architecture, Pulsar provides instant failure recovery, independent scalability and balance-free cluster expansion.</p>
+
+<p>Pulsar’s architecture follows a similar pattern to other pub-sub systems as the framework is organized in topics as the main data entity, with producers sending data to, and consumers receiving data from a topic as shown in the diagram below.</p>
+
+<center>
+<img src="/img/blog/pulsar-flink/image-1.png" width="400px" alt="Pulsar producers and consumers" />
+</center>
+<p><br /></p>
+
+<p>The second differentiator of Pulsar is that the framework is built from the get-go with <a href="https://pulsar.apache.org/docs/en/concepts-multi-tenancy/">multi-tenancy</a> in mind. What that means is that each Pulsar topic has a hierarchical management structure making the allocation of resources as well as the resource management and coordination between teams efficient and easy. With Pulsar’s multi-tenancy structure, data platform maintainers can onboar [...]
+
+<center>
+<img src="/img/blog/pulsar-flink/image-2.png" width="640px" alt="Apache Flink and Apache Pulsar" />
+</center>
+<p><br /></p>
+
+<p>Finally, Pulsar’s flexible messaging framework unifies the streaming and queuing data consumption models and provides greater flexibility. As shown in the below diagram, Pulsar holds the data in the topic while multiple teams can consume the data independently depending on their workloads and data consumption patterns.</p>
+
+<center>
+<img src="/img/blog/pulsar-flink/image-3.png" width="640px" alt="Apache Flink and Apache Pulsar" />
+</center>
+<p><br /></p>
+
+<h2 id="pulsars-view-on-data-segmented-data-streams">Pulsar’s view on data: Segmented data streams</h2>
+
+<p>Apache Flink is a streaming-first computation framework that perceives <a href="/news/2019/02/13/unified-batch-streaming-blink.html">batch processing as a special case of streaming</a>. Flink’s view on data streams distinguishes batch and stream processing between bounded and unbounded data streams, assuming that for batch workloads the data stream is finite, with a beginning and an end.</p>
+
+<p>Apache Pulsar has a similar perspective to that of Apache Flink with regards to the data layer. The framework also uses streams as a unified view on all data, while its layered architecture allows traditional pub-sub messaging for streaming workloads and continuous data processing or usage of <em>Segmented Streams</em> and bounded data stream for batch and static workloads.</p>
+
+<center>
+<img src="/img/blog/pulsar-flink/image-4.png" width="640px" alt="Apache Flink and Apache Pulsar" />
+</center>
+<p><br /></p>
+
+<p>With Pulsar, once a producer sends data to a topic, it is partitioned depending on the data traffic and then further segmented under those partitions — using Apache Bookkeeper as segment store — to allow for parallel data processing as illustrated in the diagram below. This allows a combination of traditional pub-sub messaging and distributed parallel computations in one framework.</p>
+
+<center>
+<img src="/img/blog/pulsar-flink/image-5.png" width="640px" alt="Apache Flink and Apache Pulsar" />
+</center>
+<p><br /></p>
+
+<h2 id="when-flink--pulsar-come-together">When Flink + Pulsar come together</h2>
+
+<p>Apache Flink and Apache Pulsar integrate in multiple ways already. In the following sections, I will present some potential future integrations between the frameworks and share examples of existing ways in which you can utilize the frameworks together.</p>
+
+<h3 id="potential-integrations">Potential Integrations</h3>
+
+<p>Pulsar can integrate with Apache Flink in different ways. Some potential integrations include providing support for streaming workloads with the use of <em>Streaming Connectors</em> and support for batch workloads with the use of <em>Batch Source Connectors</em>. Pulsar also comes with native support for schema that can integrate with Flink and provide structured access to the data, for example by using Flink SQL as a way of querying data in Pulsar. Final [...]
+
+<p>From an architecture point of view, we can imagine the integration between the two frameworks as one that uses Apache Pulsar for a unified view of the data layer and Apache Flink as a unified computation and data processing framework and API.</p>
+
+<h3 id="existing-integrations">Existing Integrations</h3>
+
+<p>Integration between the two frameworks is ongoing and developers can already use Pulsar with Flink in multiple ways. For example, Pulsar can be used as a streaming source and streaming sink in Flink DataStream applications. Developers can ingest data from Pulsar into a Flink job that makes computations and processes real-time data, to then send the data back to a Pulsar topic as a streaming sink. Such an example is shown below:</p>
+
+<div class="highlight"><pre><code class="language-java"><span class="c1">// create and configure Pulsar consumer</span>
+<span class="n">PulsarSourceBuilder</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span><span class="n">builder</span> <span class="o">=</span> <span class="n">PulsarSourceBuilder</span>
+ <span class="o">.</span><span class="na">builder</span><span class="o">(</span><span class="k">new</span> <span class="nf">SimpleStringSchema</span><span class="o">())</span>
+ <span class="o">.</span><span class="na">serviceUrl</span><span class="o">(</span><span class="n">serviceUrl</span><span class="o">)</span>
+ <span class="o">.</span><span class="na">topic</span><span class="o">(</span><span class="n">inputTopic</span><span class="o">)</span>
+ <span class="o">.</span><span class="na">subscriptionName</span><span class="o">(</span><span class="n">subscription</span><span class="o">);</span>
+<span class="n">SourceFunction</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">src</span> <span class="o">=</span> <span class="n">builder</span><span class="o">.</span><span class="na">build</span><span class="o"&g [...]
+<span class="c1">// ingest DataStream with Pulsar consumer</span>
+<span class="n">DataStream</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">words</span> <span class="o">=</span> <span class="n">env</span><span class="o">.</span><span class="na">addSource</span><span class="o"> [...]
+
+<span class="c1">// perform computation on DataStream (here a simple WordCount)</span>
+<span class="n">DataStream</span><span class="o">&lt;</span><span class="n">WordWithCount</span><span class="o">&gt;</span> <span class="n">wc</span> <span class="o">=</span> <span class="n">words</span>
+ <span class="o">.</span><span class="na">flatMap</span><span class="o">((</span><span class="n">FlatMapFunction</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">WordWithCount</span><span class="o">&gt;)</span> <span class=&quo [...]
+ <span class="n">collector</span><span class="o">.</span><span class="na">collect</span><span class="o">(</span><span class="k">new</span> <span class="nf">WordWithCount</span><span class="o">(</span><span class="n">word</span><span class="o">,</span> <span class="mi">1&l [...]
+ <span class="o">})</span>
+ <span class="o">.</span><span class="na">returns</span><span class="o">(</span><span class="n">WordWithCount</span><span class="o">.</span><span class="na">class</span><span class="o">)</span>
+ <span class="o">.</span><span class="na">keyBy</span><span class="o">(</span><span class="s">&quot;word&quot;</span><span class="o">)</span>
+ <span class="o">.</span><span class="na">timeWindow</span><span class="o">(</span><span class="n">Time</span><span class="o">.</span><span class="na">seconds</span><span class="o">(</span><span class="mi">5</span><span class="o">))</span>
+ <span class="o">.</span><span class="na">reduce</span><span class="o">((</span><span class="n">ReduceFunction</span><span class="o">&lt;</span><span class="n">WordWithCount</span><span class="o">&gt;)</span> <span class="o">(</span><span class="n">c1</span><span class="o&quo [...]
+ <span class="k">new</span> <span class="nf">WordWithCount</span><span class="o">(</span><span class="n">c1</span><span class="o">.</span><span class="na">word</span><span class="o">,</span> <span class="n">c1</span><span class="o">.</span><span class="na">count</span& [...]
+
+<span class="c1">// emit result via Pulsar producer</span>
+<span class="n">wc</span><span class="o">.</span><span class="na">addSink</span><span class="o">(</span><span class="k">new</span> <span class="n">FlinkPulsarProducer</span><span class="o">&lt;&gt;(</span>
+ <span class="n">serviceUrl</span><span class="o">,</span>
+ <span class="n">outputTopic</span><span class="o">,</span>
+ <span class="k">new</span> <span class="nf">AuthenticationDisabled</span><span class="o">(),</span>
+ <span class="n">wordWithCount</span> <span class="o">-&gt;</span> <span class="n">wordWithCount</span><span class="o">.</span><span class="na">toString</span><span class="o">().</span><span class="na">getBytes</span><span class="o">(</span><span class="n">UTF_8</span><span class=&quo [...]
+ <span class="n">wordWithCount</span> <span class="o">-&gt;</span> <span class="n">wordWithCount</span><span class="o">.</span><span class="na">word</span><span class="o">)</span>
+<span class="o">);</span></code></pre></div>
+
+<p>Another integration between the two frameworks that developers can take advantage of includes using Pulsar as both a streaming source and a streaming table sink for Flink SQL or Table API queries as shown in the example below:</p>
+
+<div class="highlight"><pre><code class="language-java"><span class="c1">// obtain a DataStream with words</span>
+<span class="n">DataStream</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">words</span> <span class="o">=</span> <span class="o">...</span>
+
+<span class="c1">// register DataStream as Table &quot;words&quot; with two attributes (&quot;word&quot;, &quot;ts&quot;). </span>
+<span class="c1">// &quot;ts&quot; is an event-time timestamp.</span>
+<span class="n">tableEnvironment</span><span class="o">.</span><span class="na">registerDataStream</span><span class="o">(</span><span class="s">&quot;words&quot;</span><span class="o">,</span> <span class="n">words</span><span class="o">,</span> <span class="s">&quot;word, ts.rowtime&am [...]
+
+<span class="c1">// create a TableSink that produces to Pulsar</span>
+<span class="n">TableSink</span> <span class="n">sink</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">PulsarJsonTableSink</span><span class="o">(</span>
+ <span class="n">serviceUrl</span><span class="o">,</span>
+ <span class="n">outputTopic</span><span class="o">,</span>
+ <span class="k">new</span> <span class="nf">AuthenticationDisabled</span><span class="o">(),</span>
+ <span class="n">ROUTING_KEY</span><span class="o">);</span>
+
+<span class="c1">// register Pulsar TableSink as table &quot;wc&quot;</span>
+<span class="n">tableEnvironment</span><span class="o">.</span><span class="na">registerTableSink</span><span class="o">(</span>
+ <span class="s">&quot;wc&quot;</span><span class="o">,</span>
+ <span class="n">sink</span><span class="o">.</span><span class="na">configure</span><span class="o">(</span>
+ <span class="k">new</span> <span class="n">String</span><span class="o">[]{</span><span class="s">&quot;word&quot;</span><span class="o">,</span> <span class="s">&quot;cnt&quot;</span><span class="o">},</span>
+ <span class="k">new</span> <span class="n">TypeInformation</span><span class="o">[]{</span><span class="n">Types</span><span class="o">.</span><span class="na">STRING</span><span class="o">,</span> <span class="n">Types</span><span class="o">.</span><span class="na">LONG& [...]
+
+<span class="c1">// count words per 5 seconds and write result to table &quot;wc&quot;</span>
+<span class="n">tableEnvironment</span><span class="o">.</span><span class="na">sqlUpdate</span><span class="o">(</span>
+ <span class="s">&quot;INSERT INTO wc &quot;</span> <span class="o">+</span>
+ <span class="s">&quot;SELECT word, COUNT(*) AS cnt &quot;</span> <span class="o">+</span>
+ <span class="s">&quot;FROM words &quot;</span> <span class="o">+</span>
+ <span class="s">&quot;GROUP BY word, TUMBLE(ts, INTERVAL &#39;5&#39; SECOND)&quot;</span><span class="o">);</span></code></pre></div>
+
+<p>Finally, Flink integrates with Pulsar for batch workloads as a batch sink where all results get pushed to Pulsar after Apache Flink has completed the computation in a static data set. Such an example is shown below:</p>
+
+<div class="highlight"><pre><code class="language-java"><span class="c1">// obtain DataSet from arbitrary computation</span>
+<span class="n">DataSet</span><span class="o">&lt;</span><span class="n">WordWithCount</span><span class="o">&gt;</span> <span class="n">wc</span> <span class="o">=</span> <span class="o">...</span>
+
+<span class="c1">// create PulsarOutputFormat instance</span>
+<span class="n">OutputFormat</span> <span class="n">pulsarOutputFormat</span> <span class="o">=</span> <span class="k">new</span> <span class="nf">PulsarOutputFormat</span><span class="o">(</span>
+ <span class="n">serviceUrl</span><span class="o">,</span>
+ <span class="n">topic</span><span class="o">,</span>
+ <span class="k">new</span> <span class="nf">AuthenticationDisabled</span><span class="o">(),</span>
+ <span class="n">wordWithCount</span> <span class="o">-&gt;</span> <span class="n">wordWithCount</span><span class="o">.</span><span class="na">toString</span><span class="o">().</span><span class="na">getBytes</span><span class="o">());</span>
+<span class="c1">// write DataSet to Pulsar</span>
+<span class="n">wc</span><span class="o">.</span><span class="na">output</span><span class="o">(</span><span class="n">pulsarOutputFormat</span><span class="o">);</span></code></pre></div>
+
+<h2 id="conclusion">Conclusion</h2>
+
+<p>Both Pulsar and Flink share a similar view on how the data and the computation level of an application can be <em>“streaming-first”</em> with batch as a special case streaming. With Pulsar’s Segmented Streams approach and Flink’s steps to unify batch and stream processing workloads under one framework, there are numerous ways of integrating the two technologies together to provide elastic data processing at massive scale. Subscribe to the <a href="/community. [...]
+</description>
+<pubDate>Fri, 03 May 2019 14:00:00 +0200</pubDate>
+<link>https://flink.apache.org/2019/05/03/pulsar-flink.html</link>
+<guid isPermaLink="true">/2019/05/03/pulsar-flink.html</guid>
+</item>
+
+<item>
+<title>Apache Flink's Application to Season of Docs</title>
+<description><p>The Apache Flink community is happy to announce its application to the first edition of <a href="https://developers.google.com/season-of-docs/">Season of Docs</a> by Google. The program is bringing together Open Source projects and technical writers to raise awareness for and improve documentation of Open Source projects. While the community is continuously looking for new contributors to collaborate on our documentation, we would like to take [...]
+
+<p>The community has discussed this opportunity on the <a href="https://lists.apache.org/thread.html/3c789b6187da23ad158df59bbc598543b652e3cfc1010a14e294e16a@%3Cdev.flink.apache.org%3E">dev mailinglist</a> and agreed on three project ideas to submit to the program. We have a great team of mentors (Stephan, Fabian, David, Jark &amp; Konstantin) lined up and are very much looking forward to the first proposals by potential technical writers (given we are adm [...]
+
+<h2 id="project-ideas-list">Project Ideas List</h2>
+
+<h3 id="project-1-improve-documentation-of-stream-processing-concepts">Project 1: Improve Documentation of Stream Processing Concepts</h3>
+
+<p><strong>Description:</strong> Stream processing is the processing of data in motion―in other words, computing on data directly as it is produced or received. Apache Flink has pioneered the field of distributed, stateful stream processing over the last several years. As the community has pushed the boundaries of stream processing, we have introduced new concepts that users need to become familiar with to develop and operate Apache Flink applications efficiently.
+The Apache Flink documentation [1] already contains a “concepts” section, but it is a ) incomplete and b) lacks an overall structure &amp; reading flow. In addition, “concepts”-content is also spread over the development [2] &amp; operations [3] documentation without references to the “concepts” section. An example of this can be found in [4] and [5].</p>
+
+<p>In this project, we would like to restructure, consolidate and extend the concepts documentation for Apache Flink to better guide users who want to become productive as quickly as possible. This includes better conceptual introductions to topics such as event time, state, and fault tolerance with proper linking to and from relevant deployment and development guides.</p>
+
+<p><strong>Related material:</strong></p>
+
+<ol>
+ <li><a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/">https://ci.apache.org/projects/flink/flink-docs-release-1.8/</a></li>
+ <li><a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev">https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev</a></li>
+ <li><a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/ops">https://ci.apache.org/projects/flink/flink-docs-release-1.8/ops</a></li>
+ <li><a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/concepts/programming-model.html#time">https://ci.apache.org/projects/flink/flink-docs-release-1.8/concepts/programming-model.html#time</a></li>
+ <li><a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev/event_time.html">https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev/event_time.html</a></li>
+</ol>
+
+<h3 id="project-2-improve-documentation-of-flink-deployments--operations">Project 2: Improve Documentation of Flink Deployments &amp; Operations</h3>
+
+<p><strong>Description:</strong> Stream processing is the processing of data in motion―in other words, computing on data directly as it is produced or received. Apache Flink has pioneered the field of distributed, stateful stream processing for the last few years. As a stateful distributed system in general and a continuously running, low-latency system in particular, Apache Flink deployments are non-trivial to setup and manage.
+Unfortunately, the operations [1] and monitoring documentation [2] are arguably the weakest spots of the Apache Flink documentation. While it is comprehensive and often goes into a lot of detail, it lacks an overall structure and does not address common overarching concerns of operations teams in an efficient way.</p>
+
+<p>In this project, we would like to restructure this part of the documentation and extend it if possible. Ideas for extension include: discussion of session and per-job clusters, better documentation for containerized deployments (incl. K8s), capacity planning &amp; integration into CI/CD pipelines.</p>
+
+<p><strong>Related material:</strong></p>
+
+<ol>
+ <li><a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/ops/">https://ci.apache.org/projects/flink/flink-docs-release-1.8/ops</a></li>
+ <li><a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/monitoring">https://ci.apache.org/projects/flink/flink-docs-release-1.8/monitoring</a></li>
+</ol>
+
+<h3 id="project-3-improve-documentation-for-relational-apis-table-api--sql">Project 3: Improve Documentation for Relational APIs (Table API &amp; SQL)</h3>
+
+<p><strong>Description:</strong> Apache Flink features APIs at different levels of abstraction which enables its users to trade conciseness for expressiveness. Flink’s relational APIs, SQL and the Table API, are “younger” than the DataStream and DataSet APIs, more high-level and focus on data analytics use cases. A core principle of Flink’s SQL and Table API is that they can be used to process static (batch) and continuous (streaming) data and that a program or query pr [...]
+The documentation of Flink’s relational APIs has organically grown and can be improved in a few areas. There are several on-going development efforts (e.g. Hive Integration, Python Support or Support for Interactive Programming) that aim to extend the scope of the Table API and SQL.</p>
+
+<p>The existing documentation could be reorganized to prepare for covering the new features. Moreover, it could be improved by adding a concepts section that describes the use cases and internals of the APIs in more detail. Moreover, the documentation of built-in functions could be improved by adding more concrete examples.</p>
+
+<p><strong>Related material:</strong></p>
+
+<ol>
+ <li><a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev/table">Table API &amp; SQL docs main page</a></li>
+ <li><a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev/table/functions.html">Built-in functions</a></li>
+ <li><a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev/table/common.html">Concepts</a></li>
+ <li><a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev/table/streaming/">Streaming Concepts</a></li>
+</ol>
+
+</description>
+<pubDate>Wed, 17 Apr 2019 14:00:00 +0200</pubDate>
+<link>https://flink.apache.org/news/2019/04/17/sod.html</link>
+<guid isPermaLink="true">/news/2019/04/17/sod.html</guid>
+</item>
+
+<item>
+<title>Apache Flink 1.8.0 Release Announcement</title>
+<description><p>The Apache Flink community is pleased to announce Apache Flink 1.8.0. The
+latest release includes more than 420 resolved issues and some exciting
+additions to Flink that we describe in the following sections of this post.
+Please check the <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&amp;version=12344274">complete changelog</a>
+for more details.</p>
+
+<p>Flink 1.8.0 is API-compatible with previous 1.x.y releases for APIs annotated
+with the <code>@Public</code> annotation. The release is available now and we encourage
+everyone to <a href="/downloads.html">download the release</a> and
+check out the updated
+<a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/">documentation</a>.
+Feedback through the Flink <a href="/community.html#mailing-lists">mailing
+lists</a> or
+<a href="https://issues.apache.org/jira/projects/FLINK/summary">JIRA</a> is, as always,
+very much appreciated!</p>
+
+<p>You can find the binaries on the updated <a href="/downloads.html">Downloads page</a> on the Flink project site.</p>
+
+<div class="page-toc">
+<ul id="markdown-toc">
+ <li><a href="#new-features-and-improvements" id="markdown-toc-new-features-and-improvements">New Features and Improvements</a></li>
+ <li><a href="#important-changes" id="markdown-toc-important-changes">Important Changes</a></li>
+ <li><a href="#known-issues" id="markdown-toc-known-issues">Known Issues</a></li>
+ <li><a href="#release-notes" id="markdown-toc-release-notes">Release Notes</a></li>
+ <li><a href="#list-of-contributors" id="markdown-toc-list-of-contributors">List of Contributors</a></li>
+</ul>
+
+</div>
+
+<p>With Flink 1.8.0 we come closer to our goals of enabling fast data processing
+and building data-intensive applications for the Flink community in a seamless
+way. We do this by cleaning up and refactoring Flink under the hood to allow
+more efficient feature development in the future. This includes removal of the
+legacy runtime components that were subsumed in the major rework of Flink’s
+underlying distributed system architecture
+(<a href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65147077">FLIP-6</a>)
+as well as refactorings on the Table API that prepare it for the future
+addition of the Blink enhancements
+(<a href="https://issues.apache.org/jira/browse/FLINK-11439">FLINK-11439</a>).</p>
+
+<p>Nevertheless, this release includes some important new features and bug fixes.
+The most interesting of those are highlighted below. Please consult the
+<a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&amp;version=12344274">complete changelog</a>
+and the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/release-notes/flink-1.8.html">release notes</a>
+for more details.</p>
+
+<h2 id="new-features-and-improvements">New Features and Improvements</h2>
+
+<ul>
+ <li>
+ <p><strong>Finalized State Schema Evolution Story</strong>: This release completes
+the community driven effort to provide a schema evolution story for
+user state managed by Flink. This has been an effort that spanned 2
+releases, starting from 1.7.0 with the introduction of support for
+Avro state schema evolution as well as a revamped serialization
+compatibility abstraction.</p>
+
+ <p>Flink 1.8.0 finalizes this effort by extending support for schema
+evolution to POJOs, upgrading all Flink built-in serializers to use
+the new serialization compatibility abstractions, as well as making it
+easier for advanced users who use custom state serializers to
+implement the abstractions. These different aspects for a complete
+out-of-the-box schema evolution story are explained in detail below:</p>
+
+ <ol>
+ <li>
+ <p>Support for POJO state schema evolution: The pool of data types
+that support state schema evolution has been expanded to include
+POJOs. For state types that use POJOs, you can now add or remove
+fields from your POJO while retaining backwards
+compatibility. For a full overview of the list of data types that
+now support schema evolution as well as their evolution
+specifications and limitations, please refer to the State Schema
+Evolution documentation page.</p>
+ </li>
+ <li>
+ <p>Upgrade all Flink serializers to use new serialization
+compatibility asbtractions: Back in 1.7.0, we introduced the new
+serialization compatibility abstractions <code>TypeSerializerSnapshot</code>
+and <code>TypeSerializerSchemaCompatibility</code>. Besides providing a more
+expressible API to reflect schema compatibility between the data
+stored in savepoints and the data registered at runtime, another
+important aspect about the new abstraction is that it avoids the
+need for Flink to Java-serialize the state serializer as state
+metadata in savepoints.</p>
+
+ <p>In 1.8.0, all of Flink’s built-in serializers have been upgraded to
+use the new abstractions, and therefore the serializers
+themselves are no longer Java-serialized into savepoints. This
+greatly improves interoperability of Flink savepoints, in terms
+of state schema evolvability. For example, one outcome was the
+support for POJO schema evolution, as previously mentioned
+above. Another outcome is that all composite data types supported
+by Flink (such as <code>Either</code>, Scala case classes, Flink Java
+<code>Tuple</code>s, etc.) are generally evolve-able as well when they have
+a nested evolvable type, such as a POJO. For example, the <code>MyPojo</code>
+type in <code>ValueState&lt;Tuple2&lt;Integer, MyPojo&gt;&gt;</code> or
+<code>ListState&lt;Either&lt;Integer, MyPojo&gt;&gt;</code>, which is a POJO, is allowed
+to evolve its schema.</p>
+
+ <p>For users who are using custom <code>TypeSerializer</code> implementations
+for their state serializer and are still using the outdated
+abstractions (i.e. <code>TypeSerializerConfigSnapshot</code> and
+<code>CompatiblityResult</code>), we highly recommend upgrading to the new
+abstractions to be future proof. Please refer to the Custom State
+Serialization documentation page for a detailed description on
+the new abstractions.</p>
+ </li>
+ <li>
+ <p>Provide pre-defined snapshot implementations for common
+serializers: For convenience, Flink 1.8.0 comes with two
+predefined implementations for the <code>TypeSerializerSnapshot</code> that
+make the task of implementing these new abstractions easier
+for most implementations of <code>TypeSerializer</code>s -
+<code>SimpleTypeSerializerSnapshot</code> and
+<code>CompositeTypeSerializerSnapshot</code>. This section in the
+documentation provides information on how to use these classes.</p>
+ </li>
+ </ol>
+ </li>
+ <li>
+ <p><strong>Continuous cleanup of old state based on TTL
+(<a href="https://issues.apache.org/jira/browse/FLINK-7811">FLINK-7811</a>)</strong>: We
+introduced TTL (time-to-live) for Keyed state in Flink 1.6
+(<a href="https://issues.apache.org/jira/browse/FLINK-9510">FLINK-9510</a>). This
+feature enabled cleanup and made keyed state entries inaccessible after a
+defined timeout. In addition state would now also be cleaned up when
+writing a savepoint/checkpoint.</p>
+
+ <p>Flink 1.8 introduces continuous cleanup of old entries for both the RocksDB
+state backend
+(<a href="https://issues.apache.org/jira/browse/FLINK-10471">FLINK-10471</a>) and the heap
+state backend
+(<a href="https://issues.apache.org/jira/browse/FLINK-10473">FLINK-10473</a>). This means
+that old entries (according to the TTL setting) are continuously cleaned up.</p>
+ </li>
+ <li>
+ <p><strong>SQL pattern detection with user-defined functions and
+aggregations</strong>: The support of the MATCH_RECOGNIZE clause has been
+extended by multiple features. The addition of user-defined
+functions allows for custom logic during pattern detection
+(<a href="https://issues.apache.org/jira/browse/FLINK-10597">FLINK-10597</a>),
+while adding aggregations allows for more complex CEP definitions,
+such as the following
+(<a href="https://issues.apache.org/jira/browse/FLINK-7599">FLINK-7599</a>).</p>
+
+ <div class="highlight"><pre><code>SELECT *
+FROM Ticker
+ MATCH_RECOGNIZE (
+ ORDER BY rowtime
+ MEASURES
+ AVG(A.price) AS avgPrice
+ ONE ROW PER MATCH
+ AFTER MATCH SKIP TO FIRST B
+ PATTERN (A+ B)
+ DEFINE
+ A AS AVG(A.price) &lt; 15
+ ) MR;
+</code></pre></div>
+ </li>
+ <li>
+ <p><strong>RFC-compliant CSV format (<a href="https://issues.apache.org/jira/browse/FLINK-9964">FLINK-9964</a>)</strong>: The SQL tables can now be read and written in
+an RFC-4180 standard compliant CSV table format. The format might also be
+useful for general DataStream API users.</p>
+ </li>
+ <li>
+ <p><strong>New KafkaDeserializationSchema that gives direct access to ConsumerRecord
+(<a href="https://issues.apache.org/jira/browse/FLINK-8354">FLINK-8354</a>)</strong>: For the
+Flink <code>KafkaConsumers</code>, we introduced a new <code>KafkaDeserializationSchema</code> that
+gives direct access to the Kafka <code>ConsumerRecord</code>. This now allows access to
+all data that Kafka provides for a record, including the headers. This
+subsumes the <code>KeyedSerializationSchema</code> functionality, which is deprecated but
+still available for now.</p>
+ </li>
+ <li>
+ <p><strong>Per-shard watermarking option in FlinkKinesisConsumer
+(<a href="https://issues.apache.org/jira/browse/FLINK-5697">FLINK-5697</a>)</strong>: The Kinesis
+Consumer can now emit periodic watermarks that are derived from per-shard watermarks,
+for correct event time processing with subtasks that consume multiple Kinesis shards.</p>
+ </li>
+ <li>
+ <p><strong>New consumer for DynamoDB Streams to capture table changes
+(<a href="https://issues.apache.org/jira/browse/FLINK-4582">FLINK-4582</a>)</strong>: <code>FlinkDynamoDBStreamsConsumer</code>
+is a variant of the Kinesis consumer that supports retrieval of CDC-like streams from DynamoDB tables.</p>
+ </li>
+ <li>
+ <p><strong>Support for global aggregates for subtask coordination
+(<a href="https://issues.apache.org/jira/browse/FLINK-10887">FLINK-10887</a>)</strong>:
+Designed as a solution for global source watermark tracking, <code>GlobalAggregateManager</code>
+allows sharing of information between parallel subtasks. This feature will
+be integrated into streaming connectors for watermark synchronization and
+can be used for other purposes with a user defined aggregator.</p>
+ </li>
+</ul>
+
+<h2 id="important-changes">Important Changes</h2>
+
+<ul>
+ <li>
+ <p><strong>Changes to bundling of Hadoop libraries with Flink
+(<a href="https://issues.apache.org/jira/browse/FLINK-11266">FLINK-11266</a>)</strong>:
+Convenience binaries that include hadoop are no longer released.</p>
+
+ <p>If a deployment relies on <code>flink-shaded-hadoop2</code> being included in
+<code>flink-dist</code>, then you must manually download a pre-packaged Hadoop
+jar from the optional components section of the <a href="/downloads.html">download
+page</a> and copy it into the
+<code>/lib</code> directory. Alternatively, a Flink distribution that includes
+hadoop can be built by packaging <code>flink-dist</code> and activating the
+<code>include-hadoop</code> maven profile.</p>
+
+ <p>As hadoop is no longer included in <code>flink-dist</code> by default, specifying
+<code>-DwithoutHadoop</code> when packaging <code>flink-dist</code> no longer impacts the build.</p>
+ </li>
+ <li>
+ <p><strong>FlinkKafkaConsumer will now filter restored partitions based on topic
+specification
+(<a href="https://issues.apache.org/jira/browse/FLINK-10342">FLINK-10342</a>)</strong>:
+Starting from Flink 1.8.0, the <code>FlinkKafkaConsumer</code> now always filters out
+restored partitions that are no longer associated with a specified topic to
+subscribe to in the restored execution. This behaviour did not exist in
+previous versions of the <code>FlinkKafkaConsumer</code>. If you wish to retain the
+previous behaviour, please use the
+<code>disableFilterRestoredPartitionsWithSubscribedTopics()</code> configuration method
+on the <code>FlinkKafkaConsumer</code>.</p>
+
+ <p>Consider this example: if you had a Kafka Consumer that was consuming from
+topic <code>A</code>, you did a savepoint, then changed your Kafka consumer to instead
+consume from topic <code>B</code>, and then restarted your job from the savepoint.
+Before this change, your consumer would now consume from both topic <code>A</code> and
+<code>B</code> because it was stored in state that the consumer was consuming from topic
+<code>A</code>. With the change, your consumer would only consume from topic <code>B</code> after
+restore because it now filters the topics that are stored in state using the
+configured topics.</p>
+ </li>
+ <li>
+ <p><strong>Change in the Maven modules of Table API
+(<a href="https://issues.apache.org/jira/browse/FLINK-11064">FLINK-11064</a>)</strong>: Users
+that had a <code>flink-table</code> dependency before, need to update their
+dependencies to <code>flink-table-planner</code> and the correct dependency of
+<code>flink-table-api-*</code>, depending on whether Java or Scala is used: one of
+<code>flink-table-api-java-bridge</code> or <code>flink-table-api-scala-bridge</code>.</p>
+ </li>
+</ul>
+
+<h2 id="known-issues">Known Issues</h2>
+
+<ul>
+ <li><strong>Discarded checkpoint can cause Tasks to fail
+(<a href="https://issues.apache.org/jira/browse/FLINK-11662">FLINK-11662</a>)</strong>: There is
+a race condition that can lead to erroneous checkpoint failures. This mostly
+occurs when restarting from a savepoint or checkpoint takes a long time at the
+sources of a job. If you see random checkpointing failures that don’t seem to
+have a good explanation you might be affected. Please see the Jira issue for
+more details and a workaround for the problem.</li>
+</ul>
+
+<h2 id="release-notes">Release Notes</h2>
+
+<p>Please review the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.8/release-notes/flink-1.8.html">release
+notes</a>
+for a more detailed list of changes and new features if you plan to upgrade
+your Flink setup to Flink 1.8.</p>
+
+<h2 id="list-of-contributors">List of Contributors</h2>
+
+<p>We would like to acknowledge all community members for contributing to this
+release. Special credits go to the following members for contributing to the
+1.8.0 release (according to <code>git log --pretty="%an" release-1.7.0..release-1.8.0 | sort | uniq</code> without manual deduplication):</p>
+
+<p>Addison Higham, Aitozi, Aleksey Pak, Alexander Fedulov, Alexey Trenikhin, Aljoscha Krettek, Andrey Zagrebin, Artsem Semianenka, Asura7969, Avi, Barisa Obradovic, Benchao Li, Bo WANG, Chesnay Schepler, Congxian Qiu, Cristian, David Anderson, Dawid Wysakowicz, Dian Fu, DuBin, EAlexRojas, EronWright, Eugen Yushin, Fabian Hueske, Fokko Driesprong, Gary Yao, Hequn Cheng, Igal Shilman, Jamie Grier, JaryZhen, Jeff Zhang, Jihyun Cho, Jinhu Wu, Joerg Schad, KarmaGYZ, Kezhu Wang, Konstant [...]
+
+</description>
+<pubDate>Tue, 09 Apr 2019 14:00:00 +0200</pubDate>
+<link>https://flink.apache.org/news/2019/04/09/release-1.8.0.html</link>
+<guid isPermaLink="true">/news/2019/04/09/release-1.8.0.html</guid>
+</item>
+
+<item>
+<title>Flink and Prometheus: Cloud-native monitoring of streaming applications</title>
+<description><p>This blog post describes how developers can leverage Apache Flink’s built-in <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.7/monitoring/metrics.html">metrics system</a> together with <a href="https://prometheus.io/">Prometheus</a> to observe and monitor streaming applications in an effective way. This is a follow-up post from my <a href="https://flink-forward.org/">Flink Forward</a&g [...]
+
+<h2 id="why-prometheus">Why Prometheus?</h2>
+
+<p>Prometheus is a metrics-based monitoring system that was originally created in 2012. The system is completely open-source (under the Apache License 2) with a vibrant community behind it and it has graduated from the Cloud Native Foundation last year – a sign of maturity, stability and production-readiness. As we mentioned, the system is based on metrics and it is designed to measure the overall health, behavior and performance of a service. Prometheus features a multi-dimensiona [...]
+
+<ul>
+ <li>
+ <p><strong>Metrics:</strong> Prometheus defines metrics as floats of information that change in time. These time series have millisecond precision.</p>
+ </li>
+ <li>
+ <p><strong>Labels</strong> are the key-value pairs associated with time series that support Prometheus’ flexible and powerful data model – in contrast to hierarchical data structures that one might experience with traditional metrics systems.</p>
+ </li>
+ <li>
+ <p><strong>Scrape:</strong> Prometheus is a pull-based system and fetches (“scrapes”) metrics data from specified sources that expose HTTP endpoints with a text-based format.</p>
+ </li>
+ <li>
+ <p><strong>PromQL</strong> is Prometheus’ <a href="https://prometheus.io/docs/prometheus/latest/querying/basics/">query language</a>. It can be used for both building dashboards and setting up alert rules that will trigger when specific conditions are met.</p>
+ </li>
+</ul>
+
+<p>When considering metrics and monitoring systems for your Flink jobs, there are many <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.7/monitoring/metrics.html">options</a>. Flink offers native support for exposing data to Prometheus via the <code>PrometheusReporter</code> configuration. Setting up this integration is very easy.</p>
+
+<p>Prometheus is a great choice as usually Flink jobs are not running in isolation but in a greater context of microservices. For making metrics available to Prometheus from other parts of a larger system, there are two options: There exist <a href="https://prometheus.io/docs/instrumenting/clientlibs/">libraries for all major languages</a> to instrument other applications. Additionally, there is a wide variety of <a href="https://prometheus.io/docs/ins [...]
+
+<h2 id="prometheus-and-flink-in-action">Prometheus and Flink in Action</h2>
+
+<p>We have provided a <a href="https://github.com/mbode/flink-prometheus-example">GitHub repository</a> that demonstrates the integration described above. To have a look, clone the repository, make sure <a href="https://docs.docker.com/install/">Docker</a> is installed and run:</p>
+
+<div class="highlight"><pre><code>./gradlew composeUp
+</code></pre></div>
+
+<p>This builds a Flink job using the build tool <a href="https://gradle.org/">Gradle</a> and starts up a local environment based on <a href="https://docs.docker.com/compose/">Docker Compose</a> running the job in a <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.7/ops/deployment/docker.html#flink-job-cluster">Flink job cluster</a> (reachable at <a href="http://localhost:8081/">http: [...]
+
+<center>
+<img src="/img/blog/2019-03-11-prometheus-monitoring/prometheusexamplejob.png" width="600px" alt="PrometheusExampleJob in Flink Web UI" />
+<br />
+<i><small>Job graph and custom metric for example job in Flink web interface.</small></i>
+</center>
+<p><br /></p>
+
+<p>The <code>PrometheusExampleJob</code> has three operators: Random numbers up to 10,000 are generated, then a map counts the events and creates a histogram of the values passed through. Finally, the events are discarded without further output. The very simple code below is from the second operator. It illustrates how easy it is to add custom metrics relevant to your business logic into your Flink job.</p>
+
+<div class="highlight"><pre><code class="language-java"><span class="kd">class</span> <span class="nc">FlinkMetricsExposingMapFunction</span> <span class="kd">extends</span> <span class="n">RichMapFunction</span><span class="o">&lt;</span><span class="n">Integer</span><span class="o">,</span> &l [...]
+ <span class="kd">private</span> <span class="kd">transient</span> <span class="n">Counter</span> <span class="n">eventCounter</span><span class="o">;</span>
+
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="nf">open</span><span class="o">(</span><span class="n">Configuration</span> <span class="n">parameters</span><span class="o">)</span> <span class="o">{</span>
+ <span class="n">eventCounter</span> <span class="o">=</span> <span class="n">getRuntimeContext</span><span class="o">().</span><span class="na">getMetricGroup</span><span class="o">().</span><span class="na">counter</span><span class="o">(</span><span class="s">&quot;events&quot;</spa [...]
+ <span class="o">}</span>
+
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="n">Integer</span> <span class="nf">map</span><span class="o">(</span><span class="n">Integer</span> <span class="n">value</span><span class="o">)</span> <span class="o">{</span>
+ <span class="n">eventCounter</span><span class="o">.</span><span class="na">inc</span><span class="o">();</span>
+ <span class="k">return</span> <span class="n">value</span><span class="o">;</span>
+ <span class="o">}</span>
+<span class="o">}</span></code></pre></div>
+<center><i><small>Excerpt from <a href="https://github.com/mbode/flink-prometheus-example/blob/master/src/main/java/com/github/mbode/flink_prometheus_example/FlinkMetricsExposingMapFunction.java">FlinkMetricsExposingMapFunction.java</a> demonstrating custom Flink metric.</small></i></center>
+
+<h2 id="configuring-prometheus-with-flink">Configuring Prometheus with Flink</h2>
+
+<p>To start monitoring Flink with Prometheus, the following steps are necessary:</p>
+
+<ol>
+ <li>
+ <p>Make the <code>PrometheusReporter</code> jar available to the classpath of the Flink cluster (it comes with the Flink distribution):</p>
+
+ <div class="highlight"><pre><code> cp /opt/flink/opt/flink-metrics-prometheus-1.7.2.jar /opt/flink/lib
+</code></pre></div>
+ </li>
+ <li>
+ <p><a href="https://ci.apache.org/projects/flink/flink-docs-release-1.7/monitoring/metrics.html#reporter">Configure the reporter</a> in Flink’s <em>flink-conf.yaml</em>. All job managers and task managers will expose the metrics on the configured port.</p>
+
+ <div class="highlight"><pre><code> metrics.reporters: prom
+ metrics.reporter.prom.class: org.apache.flink.metrics.prometheus.PrometheusReporter
+ metrics.reporter.prom.port: 9999
+</code></pre></div>
+ </li>
+ <li>
+ <p>Prometheus needs to know where to scrape metrics. In a static scenario, you can simply <a href="https://prometheus.io/docs/prometheus/latest/configuration/configuration/">configure Prometheus</a> in <em>prometheus.yml</em> with the following:</p>
+
+ <div class="highlight"><pre><code> scrape_configs:
+ - job_name: 'flink'
+ static_configs:
+ - targets: ['job-cluster:9999', 'taskmanager1:9999', 'taskmanager2:9999']
+</code></pre></div>
+
+ <p>In more dynamic scenarios we recommend using Prometheus’ service discovery support for different platforms such as Kubernetes, AWS EC2 and more.</p>
+ </li>
+</ol>
+
+<p>Both custom metrics are now available in Prometheus:</p>
+
+<center>
+<img src="/img/blog/2019-03-11-prometheus-monitoring/prometheus.png" width="600px" alt="Prometheus web UI with example metric" />
+<br />
+<i><small>Example metric in Prometheus web UI.</small></i>
+</center>
+<p><br /></p>
+
+<p>More technical metrics from the Flink cluster (like checkpoint sizes or duration, Kafka offsets or resource consumption) are also available. If you are interested, you can check out the HTTP endpoints exposing all Prometheus metrics for the job managers and the two task managers on <a href="http://localhost:9249/metrics">http://localhost:9249</a>, <a href="http://localhost:9250/metrics">http://localhost:9250</a> and <a href="ht [...]
+
+<p>To test Prometheus’ alerting feature, kill one of the Flink task managers via</p>
+
+<div class="highlight"><pre><code>docker kill taskmanager1
+</code></pre></div>
+
+<p>Our Flink job can recover from this partial failure via the mechanism of <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.7/dev/stream/state/checkpointing.html">Checkpointing</a>. Nevertheless, after roughly one minute (as configured in the alert rule) the following alert will fire:</p>
+
+<center>
+<img src="/img/blog/2019-03-11-prometheus-monitoring/prometheusalerts.png" width="600px" alt="Prometheus web UI with example alert" />
+<br />
+<i><small>Example alert in Prometheus web UI.</small></i>
+</center>
+<p><br /></p>
+
+<p>In real-world situations alerts like this one can be routed through a component called <a href="https://prometheus.io/docs/alerting/alertmanager/">Alertmanager</a> and be grouped into notifications to systems like email, PagerDuty or Slack.</p>
+
+<p>Go ahead and play around with the setup, and check out the <a href="https://grafana.com/grafana">Grafana</a> instance reachable at <a href="http://localhost:3000/">http://localhost:3000</a> (credentials <em>admin:flink</em>) for visualizing Prometheus metrics. If there are any questions or problems, feel free to <a href="https://github.com/mbode/flink-prometheus-example/issues">create an issue</a>. Onc [...]
+
+<div class="highlight"><pre><code>./gradlew composeDown
+</code></pre></div>
+<p><br /></p>
+
+<h2 id="conclusion">Conclusion</h2>
+
+<p>Using Prometheus together with Flink provides an easy way for effective monitoring and alerting of your Flink jobs. Both projects have exciting and vibrant communities behind them with new developments and additions scheduled for upcoming releases. We encourage you to try the two technologies together as it has immensely improved our insights into Flink jobs running in production.</p>
+</description>
+<pubDate>Mon, 11 Mar 2019 13:00:00 +0100</pubDate>
+<link>https://flink.apache.org/features/2019/03/11/prometheus-monitoring.html</link>
+<guid isPermaLink="true">/features/2019/03/11/prometheus-monitoring.html</guid>
+</item>
+
+<item>
+<title>What to expect from Flink Forward San Francisco 2019</title>
+<description><p>The third annual Flink Forward San Francisco is just a few weeks away! As always, Flink Forward will be the right place to meet and mingle with experienced Flink users, contributors, and committers. Attendees will hear and chat about the latest developments around Flink and learn from technical deep-dive sessions and exciting use cases that were put into production with Flink. The event will take place on April 1-2, 2019 at Hotel Nikko in San Francisco. The <a hr [...]
+
+<p>Some highlights of the program are:</p>
+
+<ul>
+ <li>
+ <p><a href="https://sf-2019.flink-forward.org/conference-program#realtime-store-visit-predictions-at-scale">Realtime Store Visit Predictions at Scale</a>: Luca Giovagnoli from <em>Yelp</em> will talk about a “multidisciplinary” Flink application that combines geospatial clustering algorithms, Machine Learning models, and cutting-edge stream-processing technology.</p>
+ </li>
+ <li>
+ <p><a href="https://sf-2019.flink-forward.org/conference-program#real-time-processing-with-flink-for-machine-learning-at-netflix">Real-time Processing with Flink for Machine Learning at Netflix</a>: Elliot Chow will discuss the practical aspects of using Apache Flink to power Machine Learning algorithms for video recommendations, search results ranking, and selection of artwork images at <em>Netflix</em>.</p>
+ </li>
+ <li>
+ <p><a href="https://sf-2019.flink-forward.org/conference-program#building-production-flink-jobs-with-airstream-at-airbnb">Building production Flink jobs with Airstream at Airbnb</a>: Pala Muthiah and Hao Wang will reveal how <em>Airbnb</em> builds real time data pipelines with Airstream, Airbnb’s computation framework that is powered by Flink SQL.</p>
+ </li>
+ <li>
+ <p><a href="https://sf-2019.flink-forward.org/conference-program#when-table-meets-ai--build-flink-ai-ecosystem-on-table-api">When Table meets AI: Build Flink AI Ecosystem on Table API</a>: Shaoxuan Wang from <em>Alibaba</em> will discuss how they are building a solid AI ecosystem for unified batch/streaming Machine Learning data pipelines on top of Flink’s Table API.</p>
+ </li>
+ <li>
+ <p><a href="https://sf-2019.flink-forward.org/conference-program#adventures-in-scaling-from-zero-to-5-billion-data-points-per-day">Adventures in Scaling from Zero to 5 Billion Data Points per Day</a>: Dave Torok will take us through <em>Comcast’s</em> journey in scaling the company’s operationalized Machine Learning framework from the very early days in production to processing more than 5 billion data points per day.</p>
+ </li>
+</ul>
+
+<p>If you’re new to Apache Flink or want to deepen your knowledge around the framework, Flink Forward features again a full day of training.</p>
+
+<p>You can choose from 3 training tracks:</p>
+
+<ul>
+ <li>
+ <p><a href="https://sf-2019.flink-forward.org/training-program#introduction-to-streaming-with-apache-flink">Introduction to Streaming with Apache Flink</a>: A hands-on, in-depth introduction to stream processing and Apache Flink, this course emphasizes those features of Flink that make it easy to build and manage accurate, fault tolerant applications on streams.</p>
+ </li>
+ <li>
+ <p><a href="https://sf-2019.flink-forward.org/training-program#analyzing-streaming-data-with-flink-sql">Analyzing Streaming Data with Flink SQL</a>: In this hands-on training, you will learn what it means to run SQL queries on data streams and how to fully leverage the potential of SQL on Flink. We’ll also cover some of the more recent features such as time-versioned joins and the MATCH RECOGNIZE clause.</p>
+ </li>
+ <li>
+ <p><a href="https://sf-2019.flink-forward.org/training-program#apache-flink-troubleshooting---operations">Troubleshooting and Operating Flink at large scale</a>: In this training, we will focus on everything you need to run Apache Flink applications reliably and efficiently in production including topics like capacity planning, monitoring, troubleshooting and tuning Apache Flink.</p>
+ </li>
+</ul>
+
+<p>If you haven’t done so yet, check out the <a href="http://sf-2019.flink-forward.org/conference-program">full schedule</a> and <a href="https://sf-2019.flink-forward.org/register">register</a> your attendance. <br />
+I’m looking forward to meet you at Flink Forward San Francisco.</p>
+
+<p><em>Fabian</em></p>
+</description>
+<pubDate>Wed, 06 Mar 2019 12:00:00 +0100</pubDate>
+<link>https://flink.apache.org/news/2019/03/06/ffsf-preview.html</link>
+<guid isPermaLink="true">/news/2019/03/06/ffsf-preview.html</guid>
+</item>
+
+<item>
+<title>Monitoring Apache Flink Applications 101</title>
+<description><!-- improve style of tables -->
+<style>
+ table { border: 0px solid black; table-layout: auto; width: 800px; }
+ th, td { border: 1px solid black; padding: 5px; padding-left: 10px; padding-right: 10px; }
+ th { text-align: center }
+ td { vertical-align: top }
+</style>
+
+<p>This blog post provides an introduction to Apache Flink’s built-in monitoring
+and metrics system, that allows developers to effectively monitor their Flink
+jobs. Oftentimes, the task of picking the relevant metrics to monitor a
+Flink application can be overwhelming for a DevOps team that is just starting
+with stream processing and Apache Flink. Having worked with many organizations
+that deploy Flink at scale, I would like to share my experience and some best
+practice with the community.</p>
+
+<p>With business-critical applications running on Apache Flink, performance monitoring
+becomes an increasingly important part of a successful production deployment. It
+ensures that any degradation or downtime is immediately identified and resolved
+as quickly as possible.</p>
+
+<p>Monitoring goes hand-in-hand with observability, which is a prerequisite for
+troubleshooting and performance tuning. Nowadays, with the complexity of modern
+enterprise applications and the speed of delivery increasing, an engineering
+team must understand and have a complete overview of its applications’ status at
+any given point in time.</p>
+
+<h2 id="flinks-metrics-system">Flink’s Metrics System</h2>
+
+<p>The foundation for monitoring Flink jobs is its <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.7/monitoring/metrics.html">metrics
+system</a>
+which consists of two components; <code>Metrics</code> and <code>MetricsReporters</code>.</p>
+
+<h3 id="metrics">Metrics</h3>
+
+<p>Flink comes with a comprehensive set of built-in metrics such as:</p>
+
+<ul>
+ <li>Used JVM Heap / NonHeap / Direct Memory (per Task-/JobManager)</li>
+ <li>Number of Job Restarts (per Job)</li>
+ <li>Number of Records Per Second (per Operator)</li>
+ <li>…</li>
+</ul>
+
+<p>These metrics have different scopes and measure more general (e.g. JVM or
+operating system) as well as Flink-specific aspects.</p>
+
+<p>As a user, you can and should add application-specific metrics to your
+functions. Typically these include counters for the number of invalid records or
+the number of records temporarily buffered in managed state. Besides counters,
+Flink offers additional metrics types like gauges and histograms. For
+instructions on how to register your own metrics with Flink’s metrics system
+please check out <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.7/monitoring/metrics.html#registering-metrics">Flink’s
+documentation</a>.
+In this blog post, we will focus on how to get the most out of Flink’s built-in
+metrics.</p>
+
+<h3 id="metricsreporters">MetricsReporters</h3>
+
+<p>All metrics can be queried via Flink’s REST API. However, users can configure
+MetricsReporters to send the metrics to external systems. Apache Flink provides
+reporters to the most common monitoring tools out-of-the-box including JMX,
+Prometheus, Datadog, Graphite and InfluxDB. For information about how to
+configure a reporter check out Flink’s <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.7/monitoring/metrics.html#reporter">MetricsReporter
+documentation</a>.</p>
+
+<p>In the remaining part of this blog post, we will go over some of the most
+important metrics to monitor your Apache Flink application.</p>
+
+<h2 id="monitoring-general-health">Monitoring General Health</h2>
+
+<p>The first thing you want to monitor is whether your job is actually in a <em>RUNNING</em>
+state. In addition, it pays off to monitor the number of restarts and the time
+since the last restart.</p>
+
+<p>Generally speaking, successful checkpointing is a strong indicator of the
+general health of your application. For each checkpoint, checkpoint barriers
+need to flow through the whole topology of your Flink job and events and
+barriers cannot overtake each other. Therefore, a successful checkpoint shows
+that no channel is fully congested.</p>
+
+<p><strong>Key Metrics</strong></p>
+
+<table>
+ <thead>
+ <tr>
+ <th>Metric</th>
+ <th>Scope</th>
+ <th>Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><code>uptime</code></td>
+ <td>job</td>
+ <td>The time that the job has been running without interruption.</td>
+ </tr>
+ <tr>
+ <td><code>fullRestarts</code></td>
+ <td>job</td>
+ <td>The total number of full restarts since this job was submitted.</td>
+ </tr>
+ <tr>
+ <td><code>numberOfCompletedCheckpoints</code></td>
+ <td>job</td>
+ <td>The number of successfully completed checkpoints.</td>
+ </tr>
+ <tr>
+ <td><code>numberOfFailedCheckpoints</code></td>
+ <td>job</td>
+ <td>The number of failed checkpoints.</td>
+ </tr>
+ </tbody>
+</table>
+
+<p><br /></p>
+
+<p><strong>Example Dashboard Panels</strong></p>
+
+<center>
+<img src="/img/blog/2019-02-21-monitoring-best-practices/fig-1.png" width="800px" alt="Uptime (35 minutes), Restarting Time (3 milliseconds) and Number of Full Restarts (7)" />
+<br />
+<i><small>Uptime (35 minutes), Restarting Time (3 milliseconds) and Number of Full Restarts (7)</small></i>
+</center>
+<p><br /></p>
+
+<center>
+<img src="/img/blog/2019-02-21-monitoring-best-practices/fig-2.png" width="800px" alt="Completed Checkpoints (18336), Failed (14)" />
+<br />
+<i><small>Completed Checkpoints (18336), Failed (14)</small></i>
+</center>
+<p><br /></p>
+
+<p><strong>Possible Alerts</strong></p>
+
+<ul>
+ <li><code>ΔfullRestarts</code> &gt; <code>threshold</code></li>
+ <li><code>ΔnumberOfFailedCheckpoints</code> &gt; <code>threshold</code></li>
+</ul>
+
+<h2 id="monitoring-progress--throughput">Monitoring Progress &amp; Throughput</h2>
+
+<p>Knowing that your application is RUNNING and checkpointing is working fine is good,
+but it does not tell you whether the application is actually making progress and
+keeping up with the upstream systems.</p>
+
+<h3 id="throughput">Throughput</h3>
+
+<p>Flink provides multiple metrics to measure the throughput of our application.
+For each operator or task (remember: a task can contain multiple <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.7/dev/stream/operators/#task-chaining-and-resource-groups">chained
+tasks</a>
+Flink counts the number of records and bytes going in and out. Out of those
+metrics, the rate of outgoing records per operator is often the most intuitive
+and easiest to reason about.</p>
+
+<p><strong>Key Metrics</strong></p>
+
+<table>
+ <thead>
+ <tr>
+ <th>Metric</th>
+ <th>Scope</th>
+ <th>Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><code>numRecordsOutPerSecond</code></td>
+ <td>task</td>
+ <td>The number of records this operator/task sends per second.</td>
+ </tr>
+ <tr>
+ <td><code>numRecordsOutPerSecond</code></td>
+ <td>operator</td>
+ <td>The number of records this operator sends per second.</td>
+ </tr>
+ </tbody>
+</table>
+
+<p><br /></p>
+
+<p><strong>Example Dashboard Panels</strong></p>
+
+<center>
+<img src="/img/blog/2019-02-21-monitoring-best-practices/fig-3.png" width="800px" alt="Mean Records Out per Second per Operator" />
+<br />
+<i><small>Mean Records Out per Second per Operator</small></i>
+</center>
+<p><br /></p>
+
+<p><strong>Possible Alerts</strong></p>
+
+<ul>
+ <li><code>recordsOutPerSecond</code> = <code>0</code> (for a non-Sink operator)</li>
+</ul>
+
+<p><em>Note</em>: Source operators always have zero incoming records. Sink operators
+always have zero outgoing records because the metrics only count
+Flink-internal communication. There is a <a href="https://issues.apache.org/jira/browse/FLINK-7286">JIRA
+ticket</a> to change this
+behavior.</p>
+
+<h3 id="progress">Progress</h3>
+
+<p>For applications, that use event time semantics, it is important that watermarks
+progress over time. A watermark of time <em>t</em> tells the framework, that it
+should not anymore expect to receive events with a timestamp earlier than <em>t</em>,
+and in turn, to trigger all operations that were scheduled for a timestamp &lt; <em>t</em>.
+For example, an event time window that ends at <em>t</em> = 30 will be closed and
+evaluated once the watermark passes 30.</p>
+
+<p>As a consequence, you should monitor the watermark at event time-sensitive
+operators in your application, such as process functions and windows. If the
+difference between the current processing time and the watermark, known as
+even-time skew, is unusually high, then it typically implies one of two issues.
+First, it could mean that your are simply processing old events, for example
+during catch-up after a downtime or when your job is simply not able to keep up
+and events are queuing up. Second, it could mean a single upstream sub-task has
+not sent a watermark for a long time (for example because it did not receive any
+events to base the watermark on), which also prevents the watermark in
+downstream operators to progress. This <a href="https://issues.apache.org/jira/browse/FLINK-5017">JIRA
+ticket</a> provides further
+information and a work around for the latter.</p>
+
+<p><strong>Key Metrics</strong></p>
+
+<table>
+ <thead>
+ <tr>
+ <th>Metric</th>
+ <th>Scope</th>
+ <th>Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><code>currentOutputWatermark</code></td>
+ <td>operator</td>
+ <td>The last watermark this operator has emitted.</td>
+ </tr>
+ </tbody>
+</table>
+
+<p><br /></p>
+
+<p><strong>Example Dashboard Panels</strong></p>
+
+<center>
+<img src="/img/blog/2019-02-21-monitoring-best-practices/fig-4.png" width="800px" alt="Event Time Lag per Subtask of a single operator in the topology. In this case, the watermark is lagging a few seconds behind for each subtask." />
+<br />
+<i><small>Event Time Lag per Subtask of a single operator in the topology. In this case, the watermark is lagging a few seconds behind for each subtask.</small></i>
+</center>
+<p><br /></p>
+
+<p><strong>Possible Alerts</strong></p>
+
+<ul>
+ <li><code>currentProcessingTime - currentOutputWatermark</code> &gt; <code>threshold</code></li>
+</ul>
+
+<h3 id="keeping-up">“Keeping Up”</h3>
+
+<p>When consuming from a message queue, there is often a direct way to monitor if
+your application is keeping up. By using connector-specific metrics you can
+monitor how far behind the head of the message queue your current consumer group
+is. Flink forwards the underlying metrics from most sources.</p>
+
+<p><strong>Key Metrics</strong></p>
+
+<table>
+ <thead>
+ <tr>
+ <th>Metric</th>
+ <th>Scope</th>
+ <th>Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><code>records-lag-max</code></td>
+ <td>user</td>
+ <td>applies to <code>FlinkKafkaConsumer</code>. The maximum lag in terms of the number of records for any partition in this window. An increasing value over time is your best indication that the consumer group is not keeping up with the producers.</td>
+ </tr>
+ <tr>
+ <td><code>millisBehindLatest</code></td>
+ <td>user</td>
+ <td>applies to <code>FlinkKinesisConsumer</code>. The number of milliseconds a consumer is behind the head of the stream. For any consumer and Kinesis shard, this indicates how far it is behind the current time.</td>
+ </tr>
+ </tbody>
+</table>
+
+<p><br /></p>
+
+<p><strong>Possible Alerts</strong></p>
+
+<ul>
+ <li><code>records-lag-max</code> &gt; <code>threshold</code></li>
+ <li><code>millisBehindLatest</code> &gt; <code>threshold</code></li>
+</ul>
+
+<h2 id="monitoring-latency">Monitoring Latency</h2>
+
+<p>Generally speaking, latency is the delay between the creation of an event and
+the time at which results based on this event become visible. Once the event is
+created it is usually stored in a persistent message queue, before it is
+processed by Apache Flink, which then writes the results to a database or calls
+a downstream system. In such a pipeline, latency can be introduced at each stage
+and for various reasons including the following:</p>
+
+<ol>
+ <li>It might take a varying amount of time until events are persisted in the
+message queue.</li>
+ <li>During periods of high load or during recovery, events might spend some time
+in the message queue until they are processed by Flink (see previous section).</li>
+ <li>Some operators in a streaming topology need to buffer events for some time
+(e.g. in a time window) for functional reasons.</li>
+ <li>Each computation in your Flink topology (framework or user code), as well as
+each network shuffle, takes time and adds to latency.</li>
+ <li>If the application emits through a transactional sink, the sink will only
+commit and publish transactions upon successful checkpoints of Flink, adding
+latency usually up to the checkpointing interval for each record.</li>
+</ol>
+
+<p>In practice, it has proven invaluable to add timestamps to your events at
+multiple stages (at least at creation, persistence, ingestion by Flink,
+publication by Flink, possibly sampling those to save bandwidth). The
+differences between these timestamps can be exposed as a user-defined metric in
+your Flink topology to derive the latency distribution of each stage.</p>
+
+<p>In the rest of this section, we will only consider latency, which is introduced
+inside the Flink topology and cannot be attributed to transactional sinks or
+events being buffered for functional reasons (4.).</p>
+
+<p>To this end, Flink comes with a feature called <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.7/monitoring/metrics.html#latency-tracking">Latency
+Tracking</a>.
+When enabled, Flink will insert so-called latency markers periodically at all
+sources. For each sub-task, a latency distribution from each source to this
+operator will be reported. The granularity of these histograms can be further
+controlled by setting <em>metrics.latency.granularity</em> as desired.</p>
+
+<p>Due to the potentially high number of histograms (in particular for
+<em>metrics.latency.granularity: subtask</em>), enabling latency tracking can
+significantly impact the performance of the cluster. It is recommended to only
+enable it to locate sources of latency during debugging.</p>
+
+<p><strong>Metrics</strong></p>
+
+<table>
+ <thead>
+ <tr>
+ <th>Metric</th>
+ <th>Scope</th>
+ <th>Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><code>latency</code></td>
+ <td>operator</td>
+ <td>The latency from the source operator to this operator.</td>
+ </tr>
+ <tr>
+ <td><code>restartingTime</code></td>
+ <td>job</td>
+ <td>The time it took to restart the job, or how long the current restart has been in progress.</td>
+ </tr>
+ </tbody>
+</table>
+
+<p><br /></p>
+
+<p><strong>Example Dashboard Panel</strong></p>
+
+<center>
+<img src="/img/blog/2019-02-21-monitoring-best-practices/fig-5.png" width="800px" alt="Latency distribution between a source and a single sink subtask." />
+<br />
+<i><small>Latency distribution between a source and a single sink subtask.</small></i>
+</center>
+<p><br /></p>
+
+<h2 id="jvm-metrics">JVM Metrics</h2>
+
+<p>So far we have only looked at Flink-specific metrics. As long as latency &amp;
+throughput of your application are in line with your expectations and it is
+checkpointing consistently, this is probably everything you need. On the other
+hand, if you job’s performance is starting to degrade among the firstmetrics you
+want to look at are memory consumption and CPU load of your Task- &amp; JobManager
+JVMs.</p>
+
+<h3 id="memory">Memory</h3>
+
+<p>Flink reports the usage of Heap, NonHeap, Direct &amp; Mapped memory for JobManagers
+and TaskManagers.</p>
+
+<ul>
+ <li>
+ <p>Heap memory - as with most JVM applications - is the most volatile and important
+metric to watch. This is especially true when using Flink’s filesystem
+statebackend as it keeps all state objects on the JVM Heap. If the size of
+long-living objects on the Heap increases significantly, this can usually be
+attributed to the size of your application state (check the
+<a href="https://ci.apache.org/projects/flink/flink-docs-release-1.7/monitoring/metrics.html#checkpointing">checkpointing metrics</a>
+for an estimated size of the on-heap state). The possible reasons for growing
+state are very application-specific. Typically, an increasing number of keys, a
+large event-time skew between different input streams or simply missing state
+cleanup may cause growing state.</p>
+ </li>
+ <li>
+ <p>NonHeap memory is dominated by the metaspace, the size of which is unlimited by default
+and holds class metadata as well as static content. There is a
+<a href="https://issues.apache.org/jira/browse/FLINK-10317">JIRA Ticket</a> to limit the size
+to 250 megabyte by default.</p>
+ </li>
+ <li>
+ <p>The biggest driver of Direct memory is by far the
+number of Flink’s network buffers, which can be
+<a href="https://ci.apache.org/projects/flink/flink-docs-release-1.7/ops/config.html#configuring-the-network-buffers">configured</a>.</p>
+ </li>
+ <li>
+ <p>Mapped memory is usually close to zero as Flink does not use memory-mapped files.</p>
+ </li>
+</ul>
+
+<p>In a containerized environment you should additionally monitor the overall
+memory consumption of the Job- and TaskManager containers to ensure they don’t
+exceed their resource limits. This is particularly important, when using the
+RocksDB statebackend, since RocksDB allocates a considerable amount of
+memory off heap. To understand how much memory RocksDB might use, you can
+checkout <a href="https://www.da-platform.com/blog/manage-rocksdb-memory-size-apache-flink">this blog
+post</a>
+by Stefan Richter.</p>
+
+<p><strong>Key Metrics</strong></p>
+
+<table>
+ <thead>
+ <tr>
+ <th>Metric</th>
+ <th>Scope</th>
+ <th>Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><code>Status.JVM.Memory.NonHeap.Committed</code></td>
+ <td>job-/taskmanager</td>
+ <td>The amount of non-heap memory guaranteed to be available to the JVM (in bytes).</td>
+ </tr>
+ <tr>
+ <td><code>Status.JVM.Memory.Heap.Used</code></td>
+ <td>job-/taskmanager</td>
+ <td>The amount of heap memory currently used (in bytes).</td>
+ </tr>
+ <tr>
+ <td><code>Status.JVM.Memory.Heap.Committed</code></td>
+ <td>job-/taskmanager</td>
+ <td>The amount of heap memory guaranteed to be available to the JVM (in bytes).</td>
+ </tr>
+ <tr>
+ <td><code>Status.JVM.Memory.Direct.MemoryUsed</code></td>
+ <td>job-/taskmanager</td>
+ <td>The amount of memory used by the JVM for the direct buffer pool (in bytes).</td>
+ </tr>
+ <tr>
+ <td><code>Status.JVM.Memory.Mapped.MemoryUsed</code></td>
+ <td>job-/taskmanager</td>
+ <td>The amount of memory used by the JVM for the mapped buffer pool (in bytes).</td>
+ </tr>
+ <tr>
+ <td><code>Status.JVM.GarbageCollector.G1 Young Generation.Time</code></td>
+ <td>job-/taskmanager</td>
+ <td>The total time spent performing G1 Young Generation garbage collection.</td>
+ </tr>
+ <tr>
+ <td><code>Status.JVM.GarbageCollector.G1 Old Generation.Time</code></td>
+ <td>job-/taskmanager</td>
+ <td>The total time spent performing G1 Old Generation garbage collection.</td>
+ </tr>
+ </tbody>
+</table>
+
+<p><br /></p>
+
+<p><strong>Example Dashboard Panel</strong></p>
+
+<center>
+<img src="/img/blog/2019-02-21-monitoring-best-practices/fig-6.png" width="800px" alt="TaskManager memory consumption and garbage collection times." />
+<br />
+<i><small>TaskManager memory consumption and garbage collection times.</small></i>
+</center>
+<p><br /></p>
+
+<center>
+<img src="/img/blog/2019-02-21-monitoring-best-practices/fig-7.png" width="800px" alt="JobManager memory consumption and garbage collection times." />
+<br />
+<i><small>JobManager memory consumption and garbage collection times.</small></i>
+</center>
+<p><br /></p>
+
+<p><strong>Possible Alerts</strong></p>
+
+<ul>
+ <li><code>container memory limit</code> &lt; <code>container memory + safety margin</code></li>
+</ul>
+
+<h3 id="cpu">CPU</h3>
+
+<p>Besides memory, you should also monitor the CPU load of the TaskManagers. If
+your TaskManagers are constantly under very high load, you might be able to
+improve the overall performance by decreasing the number of task slots per
+TaskManager (in case of a Standalone setup), by providing more resources to the
+TaskManager (in case of a containerized setup), or by providing more
+TaskManagers. In general, a system already running under very high load during
+normal operations, will need much more time to catch-up after recovering from a
+downtime. During this time you will see a much higher latency (event-time skew) than
+usual.</p>
+
+<p>A sudden increase in the CPU load might also be attributed to high garbage
+collection pressure, which should be visible in the JVM memory metrics as well.</p>
+
+<p>If one or a few TaskManagers are constantly under very high load, this can slow
+down the whole topology due to long checkpoint alignment times and increasing
+event-time skew. A common reason is skew in the partition key of the data, which
+can be mitigated by pre-aggregating before the shuffle or keying on a more
+evenly distributed key.</p>
+
+<p><strong>Key Metrics</strong></p>
+
+<table>
+ <thead>
+ <tr>
+ <th>Metric</th>
+ <th>Scope</th>
+ <th>Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><code>Status.JVM.CPU.Load</code></td>
+ <td>job-/taskmanager</td>
+ <td>The recent CPU usage of the JVM.</td>
+ </tr>
+ </tbody>
+</table>
+
+<p><br /></p>
+
+<p><strong>Example Dashboard Panel</strong></p>
+
+<center>
+<img src="/img/blog/2019-02-21-monitoring-best-practices/fig-8.png" width="800px" alt="TaskManager &amp; JobManager CPU load." />
+<br />
+<i><small>TaskManager &amp; JobManager CPU load.</small></i>
+</center>
+<p><br /></p>
+
+<h2 id="system-resources">System Resources</h2>
+
+<p>In addition to the JVM metrics above, it is also possible to use Flink’s metrics
+system to gather insights about system resources, i.e. memory, CPU &amp;
+network-related metrics for the whole machine as opposed to the Flink processes
+alone. System resource monitoring is disabled by default and requires additional
+dependencies on the classpath. Please check out the
+<a href="https://ci.apache.org/projects/flink/flink-docs-release-1.7/monitoring/metrics.html#system-resources">Flink system resource metrics documentation</a> for
+additional guidance and details. System resource monitoring in Flink can be very
+helpful in setups without existing host monitoring capabilities.</p>
+
+<h2 id="conclusion">Conclusion</h2>
+
+<p>This post tries to shed some light on Flink’s metrics and monitoring system. You
+can utilise it as a starting point when you first think about how to
+successfully monitor your Flink application. I highly recommend to start
+monitoring your Flink application early on in the development phase. This way
+you will be able to improve your dashboards and alerts over time and, more
+importantly, observe the performance impact of the changes to your application
+throughout the development phase. By doing so, you can ask the right questions
+about the runtime behaviour of your application, and learn much more about
+Flink’s internals early on.</p>
+
+<p>Last but not least, this post only scratches the surface of the overall metrics
+and monitoring capabilities of Apache Flink. I highly recommend going over
+<a href="https://ci.apache.org/projects/flink/flink-docs-release-1.7/monitoring/metrics.html">Flink’s metrics documentation</a>
+for a full reference of Flink’s metrics system.</p>
+</description>
+<pubDate>Mon, 25 Feb 2019 13:00:00 +0100</pubDate>
+<link>https://flink.apache.org/news/2019/02/25/monitoring-best-practices.html</link>
+<guid isPermaLink="true">/news/2019/02/25/monitoring-best-practices.html</guid>
+</item>
+
+<item>
+<title>Apache Flink 1.6.4 Released</title>
+<description><p>The Apache Flink community released the fourth bugfix version of the Apache Flink 1.6 series.</p>
+
+<p>This release includes more than 25 fixes and minor improvements for Flink 1.6.3. The list below includes a detailed list of all fixes.</p>
+
+<p>We highly recommend all users to upgrade to Flink 1.6.4.</p>
+
+<p>Updated Maven dependencies:</p>
+
+<div class="highlight"><pre><code class="language-xml"><span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-java<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.6.4<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span>
+<span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-streaming-java_2.11<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.6.4<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span>
+<span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-clients_2.11<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.6.4<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span></code></pre></div>
+
+<p>You can find the binaries on the updated <a href="/downloads.html">Downloads page</a>.</p>
+
+<p>List of resolved issues:</p>
+
+<h2> Bug
+</h2>
+<ul>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10721">FLINK-10721</a>] - Kafka discovery-loop exceptions may be swallowed
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10761">FLINK-10761</a>] - MetricGroup#getAllVariables can deadlock
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10774">FLINK-10774</a>] - connection leak when partition discovery is disabled and open throws exception
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10848">FLINK-10848</a>] - Flink&#39;s Yarn ResourceManager can allocate too many excess containers
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11022">FLINK-11022</a>] - Update LICENSE and NOTICE files for older releases
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11071">FLINK-11071</a>] - Dynamic proxy classes cannot be resolved when deserializing job graph
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11084">FLINK-11084</a>] - Incorrect ouput after two consecutive split and select
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11119">FLINK-11119</a>] - Incorrect Scala example for Table Function
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11134">FLINK-11134</a>] - Invalid REST API request should not log the full exception in Flink logs
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11151">FLINK-11151</a>] - FileUploadHandler stops working if the upload directory is removed
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11173">FLINK-11173</a>] - Proctime attribute validation throws an incorrect exception message
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11224">FLINK-11224</a>] - Log is missing in scala-shell
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11232">FLINK-11232</a>] - Empty Start Time of sub-task on web dashboard
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11234">FLINK-11234</a>] - ExternalTableCatalogBuilder unable to build a batch-only table
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11235">FLINK-11235</a>] - Elasticsearch connector leaks threads if no connection could be established
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11251">FLINK-11251</a>] - Incompatible metric name on prometheus reporter
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11389">FLINK-11389</a>] - Incorrectly use job information when call getSerializedTaskInformation in class TaskDeploymentDescriptor
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11584">FLINK-11584</a>] - ConfigDocsCompletenessITCase fails DescriptionBuilder#linebreak() is used
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11585">FLINK-11585</a>] - Prefix matching in ConfigDocsGenerator can result in wrong assignments
+</li>
+</ul>
+
+<h2> Improvement
+</h2>
+<ul>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10910">FLINK-10910</a>] - Harden Kubernetes e2e test
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11079">FLINK-11079</a>] - Skip deployment for flnk-storm-examples
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11207">FLINK-11207</a>] - Update Apache commons-compress from 1.4.1 to 1.18
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11262">FLINK-11262</a>] - Bump jython-standalone to 2.7.1
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11289">FLINK-11289</a>] - Rework example module structure to account for licensing
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11304">FLINK-11304</a>] - Typo in time attributes doc
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11469">FLINK-11469</a>] - fix Tuning Checkpoints and Large State doc
+</li>
+</ul>
+</description>
+<pubDate>Mon, 25 Feb 2019 01:00:00 +0100</pubDate>
+<link>https://flink.apache.org/news/2019/02/25/release-1.6.4.html</link>
+<guid isPermaLink="true">/news/2019/02/25/release-1.6.4.html</guid>
+</item>
+
+<item>
+<title>Apache Flink 1.7.2 Released</title>
+<description><p>The Apache Flink community released the second bugfix version of the Apache Flink 1.7 series.</p>
+
+<p>This release includes more than 40 fixes and minor improvements for Flink 1.7.1, covering several critical
+recovery issues as well as problems in the Flink streaming connectors.</p>
+
+<p>The list below includes a detailed list of all fixes.
+We highly recommend all users to upgrade to Flink 1.7.2.</p>
+
+<p>Updated Maven dependencies:</p>
+
+<div class="highlight"><pre><code class="language-xml"><span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-java<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.7.2<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span>
+<span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-streaming-java_2.11<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.7.2<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span>
+<span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-clients_2.11<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.7.2<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span></code></pre></div>
+
+<p>You can find the binaries on the updated <a href="/downloads.html">Downloads page</a>.</p>
+
+<p>List of resolved issues:</p>
+
+<h2> Sub-task
+</h2>
+<ul>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11179">FLINK-11179</a>] - JoinCancelingITCase#testCancelSortMatchWhileDoingHeavySorting test error
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11180">FLINK-11180</a>] - ProcessFailureCancelingITCase#testCancelingOnProcessFailure
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11181">FLINK-11181</a>] - SimpleRecoveryITCaseBase test error
+</li>
+</ul>
+
+<h2> Bug
+</h2>
+<ul>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10721">FLINK-10721</a>] - Kafka discovery-loop exceptions may be swallowed
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10761">FLINK-10761</a>] - MetricGroup#getAllVariables can deadlock
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10774">FLINK-10774</a>] - connection leak when partition discovery is disabled and open throws exception
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10848">FLINK-10848</a>] - Flink&#39;s Yarn ResourceManager can allocate too many excess containers
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11046">FLINK-11046</a>] - ElasticSearch6Connector cause thread blocked when index failed with retry
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11071">FLINK-11071</a>] - Dynamic proxy classes cannot be resolved when deserializing job graph
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11083">FLINK-11083</a>] - CRowSerializerConfigSnapshot is not instantiable
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11084">FLINK-11084</a>] - Incorrect ouput after two consecutive split and select
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11100">FLINK-11100</a>] - Presto S3 FileSystem E2E test broken
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11119">FLINK-11119</a>] - Incorrect Scala example for Table Function
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11134">FLINK-11134</a>] - Invalid REST API request should not log the full exception in Flink logs
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11145">FLINK-11145</a>] - Fix Hadoop version handling in binary release script
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11151">FLINK-11151</a>] - FileUploadHandler stops working if the upload directory is removed
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11168">FLINK-11168</a>] - LargePlanTest times out on Travis
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11173">FLINK-11173</a>] - Proctime attribute validation throws an incorrect exception message
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11187">FLINK-11187</a>] - StreamingFileSink with S3 backend transient socket timeout issues
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11191">FLINK-11191</a>] - Exception in code generation when ambiguous columns in MATCH_RECOGNIZE
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11194">FLINK-11194</a>] - missing Scala 2.12 build of HBase connector
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11201">FLINK-11201</a>] - Document SBT dependency requirements when using MiniClusterResource
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11224">FLINK-11224</a>] - Log is missing in scala-shell
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11227">FLINK-11227</a>] - The DescriptorProperties contains some bounds checking errors
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11232">FLINK-11232</a>] - Empty Start Time of sub-task on web dashboard
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11234">FLINK-11234</a>] - ExternalTableCatalogBuilder unable to build a batch-only table
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11235">FLINK-11235</a>] - Elasticsearch connector leaks threads if no connection could be established
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11246">FLINK-11246</a>] - Fix distinct AGG visibility issues
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11251">FLINK-11251</a>] - Incompatible metric name on prometheus reporter
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11279">FLINK-11279</a>] - Invalid week interval parsing in ExpressionParser
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11302">FLINK-11302</a>] - FlinkS3FileSystem uses an incorrect path for temporary files.
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11389">FLINK-11389</a>] - Incorrectly use job information when call getSerializedTaskInformation in class TaskDeploymentDescriptor
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11419">FLINK-11419</a>] - StreamingFileSink fails to recover after taskmanager failure
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11436">FLINK-11436</a>] - Java deserialization failure of the AvroSerializer when used in an old CompositeSerializers
+</li>
+</ul>
+
+<h2> New Feature
+</h2>
+<ul>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10457">FLINK-10457</a>] - Support SequenceFile for StreamingFileSink
+</li>
+</ul>
+
+<h2> Improvement
+</h2>
+<ul>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10910">FLINK-10910</a>] - Harden Kubernetes e2e test
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11023">FLINK-11023</a>] - Update LICENSE and NOTICE files for flink-connectors
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11079">FLINK-11079</a>] - Skip deployment for flink-storm-examples
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11207">FLINK-11207</a>] - Update Apache commons-compress from 1.4.1 to 1.18
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11216">FLINK-11216</a>] - Back to top button is missing in the Joining document and is not properly placed in the Process Function document
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11262">FLINK-11262</a>] - Bump jython-standalone to 2.7.1
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11289">FLINK-11289</a>] - Rework example module structure to account for licensing
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11304">FLINK-11304</a>] - Typo in time attributes doc
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11331">FLINK-11331</a>] - Fix errors in tableApi.md and functions.md
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11469">FLINK-11469</a>] - fix Tuning Checkpoints and Large State doc
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11473">FLINK-11473</a>] - Clarify Documenation on Latency Tracking
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11628">FLINK-11628</a>] - Cache maven on travis
+</li>
+</ul>
+</description>
+<pubDate>Fri, 15 Feb 2019 13:00:00 +0100</pubDate>
+<link>https://flink.apache.org/news/2019/02/15/release-1.7.2.html</link>
+<guid isPermaLink="true">/news/2019/02/15/release-1.7.2.html</guid>
+</item>
+
+<item>
+<title>Batch as a Special Case of Streaming and Alibaba's contribution of Blink</title>
+<description><p>Last week, we <a href="https://lists.apache.org/thread.html/2f7330e85d702a53b4a2b361149930b50f2e89d8e8a572f8ee2a0e6d@%3Cdev.flink.apache.org%3E">broke the news</a> that Alibaba decided to contribute its Flink-fork, called Blink, back to the Apache Flink project. Why is that a big thing for Flink, what will it mean for users and the community, and how does it fit into Flink’s overall vision? Let’s take a step back to understand this better…</p>
+
+<h2 id="a-unified-approach-to-batch-and-streaming">A Unified Approach to Batch and Streaming</h2>
+
+<p>Since its early days, Apache Flink has followed the philosophy of taking a unified approach to batch and streaming data processing. The core building block is <em>“continuous processing of unbounded data streams”</em>: if you can do that, you can also do offline processing of bounded data sets (batch processing use cases), because these are just streams that happen to end at some point.</p>
+
+<center>
+<img src="/img/blog/unified-batch-streaming-blink/bounded-unbounded.png" width="600px" alt="Processing of bounded and unbounded data." />
+</center>
+<p><br /></p>
+
+<p>The <em>“streaming first, with batch as a special case of streaming”</em> philosophy is supported by various projects (for example <a href="https://flink.apache.org">Flink</a>, <a href="https://beam.apache.org">Beam</a>, etc.) and often been cited as a powerful way to build data applications that <a href="https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-101">generalize across real-time and offl [...]
+
+<h3 id="why-are-there-still-batch-processors">Why are there still batch processors?</h3>
+
+<p>However, <em>“batch is just a special case of streaming”</em> does not mean that any stream processor is now the right tool for your batch processing use cases - the introduction of stream processors did not render batch processors obsolete:</p>
+
+<ul>
+ <li>
+ <p>Pure stream processing systems are very slow at batch processing workloads. No one would consider it a good idea to use a stream processor that shuffles through message queues to analyze large amounts of available data.</p>
+ </li>
+ <li>
+ <p>Unified APIs like <a href="https://beam.apache.org">Apache Beam</a> often delegate to different runtimes depending on whether the data is continuous/unbounded of fix/bounded. For example, the implementations of the batch and streaming runtime of Google Cloud Dataflow are different, to get the desired performance and resilience in each case.</p>
+ </li>
+ <li>
+ <p><em>Apache Flink</em> has a streaming API that can do bounded/unbounded use cases, but still offers a separate DataSet API and runtime stack that is faster for batch use cases.</p>
+ </li>
+</ul>
+
+<p>What is the reason for the above? Where did <em>“batch is just a special case of streaming”</em> go wrong?</p>
+
+<p>The answer is simple, nothing is wrong with that paradigm. Unifying batch and streaming in the API is one aspect. One needs to also exploit certain characteristics of the special case “bounded data” in the runtime to competitively handle batch processing use cases. After all, batch processors have been built specifically for that special case.</p>
+
+<h2 id="batch-on-top-of-a-streaming-runtime">Batch on top of a Streaming Runtime</h2>
+
+<p>We always believed that it is possible to have a runtime that is state-of-the-art for both stream processing and batch processing use cases at the same time. A runtime that is streaming-first, but can exploit just the right amount of special properties of bounded streams to be as fast for batch use cases as dedicated batch processors. <strong>This is the unique approach that Flink takes.</strong></p>
+
+<p>Apache Flink has a network stack that supports both <a href="https://www.ververica.com/flink-forward-berlin/resources/improving-throughput-and-latency-with-flinks-network-stack">low-latency/high-throughput streaming data exchanges</a>, as well as high-throughput batch shuffles. Flink has streaming runtime operators for many operations, but also specialized operators for bounded inputs, which get used when you choose the DataSet API or select the batch envir [...]
+
+<center>
+<img src="/img/blog/unified-batch-streaming-blink/stream-batch-joins.png" width="500px" alt="Streaming and batch joins." />
+<br />
+<i>The figure illustrates a streaming join and a batch join. The batch join can read one input fully into a hash table and then probe with the other input. The stream join needs to build tables for both sides, because it needs to continuously process both inputs.
+For data larger than memory, the batch join can partition both data sets into subsets that fit in memory (data hits disk once) whereas the continuous nature of the stream join requires it to always keep all data in the table and repeatedly hit disk on cache misses.</i>
+</center>
+<p><br /></p>
+
+<p>Because of that, Apache Flink has been actually demonstrating some pretty impressive batch processing performance since its early days. The below benchmark is a bit older, but validated our architectural approach early on.</p>
+
+<center>
+<img src="/img/blog/unified-batch-streaming-blink/sort-performance.png" width="500px" alt="Sort performance." />
+<br />
+<i>Time to sort 3.2 TB (80 GB/node), in seconds<br />
+(<a href="https://www.slideshare.net/FlinkForward/dongwon-kim-a-comparative-performance-evaluation-of-flink" target="blank">Presentation by Dongwon Kim, Flink Forward Berlin 2015</a>.)</i>
+</center>
+<p><br /></p>
+
+<h2 id="what-is-still-missing">What is still missing?</h2>
+
+<p>To conclude the approach and make Flink’s experience on bounded data (batch) state-of-the-art, we need to add a few more enhancements. We believe that these features are key to realizing our vision:</p>
+
+<p><strong>(1) A truly unified runtime operator stack</strong>: Currently the bounded and unbounded operators have a different network and threading model and don’t mix and match. The original reason was that batch operators followed a “pull model” (easier for batch algorithms), while streaming operators followed a “push model” (better latency/throughput characteristics). In a unified stack, continuous streaming operators are the foundation. When operating on bounded da [...]
+
+<p><strong>(2) Exploiting bounded streams to reduce the scope of fault tolerance</strong>: When input data is bounded, it is possible to completely buffer data during shuffles (memory or disk) and replay that data after a failure. This makes recovery more fine grained and thus much more efficient.</p>
+
+<p><strong>(3) Exploiting bounded stream operator properties for scheduling</strong>: A continuous unbounded streaming application needs (by definition) all operators running at the same time. An application on bounded data can schedule operations after another, depending on how the operators consume data (e.g., first build hash table, then probe hash table). This increases resource efficiency.</p>
+
+<p><strong>(4) Enabling these special case optimizations for the DataStream API</strong>: Currently, only the Table API (which is unified across bounded/unbounded streams) activates these optimizations when working on bounded data.</p>
+
+<p><strong>(5) Performance and coverage for SQL</strong>: SQL is the de-facto standard data language, and while it is also being rapidly adopted for continuous streaming use cases, there is absolutely no way past it for bounded/batch use cases. To be competitive with the best batch engines, Flink needs more coverage and performance for the SQL query execution. While the core data-plane in Flink is high performance, the speed of SQL execution ultimately depends a lot als [...]
+
+<h2 id="enter-blink">Enter Blink</h2>
+
+<p>Blink is a fork of Apache Flink, originally created inside Alibaba to improve Flink’s behavior for internal use cases. Blink adds a series of improvements and integrations (see the <a href="https://github.com/apache/flink/blob/blink/README.md">Readme</a> for details), many of which fall into the category of improved bounded-data/batch processing and SQL. In fact, of the above list of features for a unified batch/streaming system, Blink implements significan [...]
+
+<p><strong>Unified Stream Operators:</strong> Blink extends the Flink streaming runtime operator model to support selectively reading from different inputs, while keeping the push model for very low latency. This control over the inputs helps to now support algorithms like hybrid hash-joins on the same operator and threading model as continuous symmetric joins through RocksDB. These operators also form the basis for future features like <a href="https://cwiki.ap [...]
+
+<p><strong>Table API &amp; SQL Query Processor:</strong> The SQL query processor is the component that evolved the changed most compared to the latest Flink master branch:</p>
+
+<ul>
+ <li>
+ <p>While Flink currently translates queries either into DataSet or DataStream programs (depending on the characteristics of their inputs), Blink translates queries to a data flow of the aforementioned stream operators.</p>
+ </li>
+ <li>
+ <p>Blink adds many more runtime operators for common SQL operations like semi-joins, anti-joins, etc.</p>
+ </li>
+ <li>
+ <p>The query planner (optimizer) is still based on Apache Calcite, but has many more optimization rules (incl. join reordering) and uses a proper cost model for planning.</p>
+ </li>
+ <li>
+ <p>Stream operators are more aggressively chained.</p>
+ </li>
+ <li>
+ <p>The common data structures (sorters, hash tables) and serializers are extended to go even further in operating on binary data and saving serialization overhead. Code generation is used for the row serializers.</p>
+ </li>
+</ul>
+
+<p><strong>Improved Scheduling and Failure Recovery:</strong> Finally, Blink implements several improvements for task scheduling and fault tolerance. The scheduling strategies use resources better by exploiting how the operators process their input data. The failover strategies recover more fine-grained along the boundaries of persistent shuffles. A failed JobManager can be replaced without restarting a running application.</p>
+
+<p>The changes in Blink result in a big improvement in performance. The below numbers were reported by the developers of Blink to give a rough impression of the performance gains.</p>
+
+<center>
+<img src="/img/blog/unified-batch-streaming-blink/blink-flink-tpch.png" width="600px" alt="TPC-H performance of Blink and Flink." />
+<br />
+<i>Relative performance of Blink versus Flink 1.6.0 in the TPC-H benchmark, query by query.<br />
+The performance improvement is in average 10x.<br />
+<a href="https://www.ververica.com/flink-forward-berlin/resources/unified-engine-for-data-processing-and-ai" target="blank">Presentation by Xiaowei Jiang at Flink Forward Berlin, 2018</a>.)</i>
+</center>
+<p><br /></p>
+
+<center>
+<img src="/img/blog/unified-batch-streaming-blink/blink-spark-tpcds.png" width="600px" alt="TPC-DS performace of Blink and Spark." />
+<br />
+<i>Performance of Blink versus Spark in the TPC-DS benchmark, aggregate time for all queries together.<br />
+<a href="https://www.bilibili.com/video/av42325467/?p=3" target="blank">Presentation by Xiaowei Jiang at Flink Forward Beijing, 2018</a>.</i>
+</center>
+<p><br /></p>
+
+<h2 id="how-do-we-plan-to-merge-blink-and-flink">How do we plan to merge Blink and Flink?</h2>
+
+<p>Blink’s code is currently available as a <a href="https://github.com/apache/flink/tree/blink">branch</a> in the Apache Flink repository. It is a challenge to merge a such big amount of changes, while making the merge process as non-disruptive as possible and keeping public APIs as stable as possible.</p>
+
+<p>The community’s <a href="https://lists.apache.org/thread.html/6066abd0f09fc1c41190afad67770ede8efd0bebc36f00938eecc118@%3Cdev.flink.apache.org%3E">merge plan</a> focuses initially on the bounded/batch processing features mentioned above and follows the following approach to ensure a smooth integration:</p>
+
+<ul>
+ <li>
+ <p>To merge Blink’s <em>SQL/Table API query processor</em> enhancements, we exploit the fact that both Flink and Blink have the same APIs: SQL and the Table API.
+Following some restructuring of the Table/SQL module (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions">FLIP-32</a>) we plan to merge the Blink query planner (optimizer) and runtime (operators) as an additional query processor next to the current SQL runtime. Think of it as two different runners for the same APIs.<br />
+Initially, users will be able to select which query processor to use. After a transition period in which the new query processor will be developed to subsume the current query processor, the current processor will most likely be deprecated and eventually dropped. Given that SQL is such a well defined interface, we anticipate that this transition has little friction for users. Mostly a pleasant surprise to have broader SQL feature coverage and a boost in performance.</p>
+ </li>
+ <li>
+ <p>To support the merge of Blink’s <em>enhancements to scheduling and recovery</em> for jobs on bounded data, the Flink community is already working on refactoring its current schedule and adding support for <a href="https://issues.apache.org/jira/browse/FLINK-10429">pluggable scheduling and fail-over strategies</a>.<br />
+Once this effort is finished, we can add Blink’s scheduling and recovery strategies as a new scheduling strategy that is used by the new query processor. Eventually, we plan to use the new scheduling strategy also for bounded DataStream programs.</p>
+ </li>
+ <li>
+ <p>The extended catalog support, DDL support, as well as support for Hive’s catalog and integrations is currently going through separate design discussions. We plan to leverage existing code here whenever it makes sense.</p>
+ </li>
+</ul>
+
+<h2 id="summary">Summary</h2>
+
+<p>We believe that the data processing stack of the future is based on stream processing: The elegance of stream processing with its ability to model offline processing (batch), real-time data processing, and event-driven applications in the same way, while offering high performance and consistency is simply too compelling.</p>
+
+<p>Exploiting certain properties of bounded data is important for a stream processor to achieve the same performance as dedicated batch processors. While Flink always supported batch processing, the project is taking the next step in building a unified runtime and towards <strong>becoming a stream processor that is competitive with batch processing systems even on their home turf: OLAP SQL.</strong> The contribution of Alibaba’s Blink code helps the Flink community to p [...]
+</description>
+<pubDate>Wed, 13 Feb 2019 13:00:00 +0100</pubDate>
+<link>https://flink.apache.org/news/2019/02/13/unified-batch-streaming-blink.html</link>
+<guid isPermaLink="true">/news/2019/02/13/unified-batch-streaming-blink.html</guid>
+</item>
+
+<item>
+<title>Apache Flink 1.5.6 Released</title>
+<description><p>The Apache Flink community released the sixth and last bugfix version of the Apache Flink 1.5 series.</p>
+
+<p>This release includes more than 47 fixes and minor improvements for Flink 1.5.5. The list below includes a detailed list of all fixes.</p>
+
+<p>We highly recommend all users to upgrade to Flink 1.5.6.</p>
+
+<p>Updated Maven dependencies:</p>
+
+<div class="highlight"><pre><code class="language-xml"><span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-java<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.5.6<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span>
+<span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-streaming-java_2.11<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.5.6<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span>
+<span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-clients_2.11<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.5.6<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span></code></pre></div>
+
+<p>You can find the binaries on the updated <a href="http://flink.apache.org/downloads.html">Downloads page</a>.</p>
+
+<p>List of resolved issues:</p>
+
+<h2> Sub-task
+</h2>
+<ul>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10252">FLINK-10252</a>] - Handle oversized metric messages
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10863">FLINK-10863</a>] - Assign uids to all operators
+</li>
+</ul>
+
+<h2> Bug
+</h2>
+<ul>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-8336">FLINK-8336</a>] - YarnFileStageTestS3ITCase.testRecursiveUploadForYarnS3 test instability
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-9646">FLINK-9646</a>] - ExecutionGraphCoLocationRestartTest.testConstraintsAfterRestart failed on Travis
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10166">FLINK-10166</a>] - Dependency problems when executing SQL query in sql-client
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10309">FLINK-10309</a>] - Cancel with savepoint fails with java.net.ConnectException when using the per job-mode
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10419">FLINK-10419</a>] - ClassNotFoundException while deserializing user exceptions from checkpointing
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10455">FLINK-10455</a>] - Potential Kafka producer leak in case of failures
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10482">FLINK-10482</a>] - java.lang.IllegalArgumentException: Negative number of in progress checkpoints
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10491">FLINK-10491</a>] - Deadlock during spilling data in SpillableSubpartition
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10566">FLINK-10566</a>] - Flink Planning is exponential in the number of stages
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10581">FLINK-10581</a>] - YarnConfigurationITCase.testFlinkContainerMemory test instability
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10642">FLINK-10642</a>] - CodeGen split fields errors when maxGeneratedCodeLength equals 1
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10655">FLINK-10655</a>] - RemoteRpcInvocation not overwriting ObjectInputStream&#39;s ClassNotFoundException
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10669">FLINK-10669</a>] - Exceptions &amp; errors are not properly checked in logs in e2e tests
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10670">FLINK-10670</a>] - Fix Correlate codegen error
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10674">FLINK-10674</a>] - Fix handling of retractions after clean up
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10690">FLINK-10690</a>] - Tests leak resources via Files.list
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10693">FLINK-10693</a>] - Fix Scala EitherSerializer duplication
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10715">FLINK-10715</a>] - E2e tests fail with ConcurrentModificationException in MetricRegistryImpl
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10750">FLINK-10750</a>] - SocketClientSinkTest.testRetry fails on Travis
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10752">FLINK-10752</a>] - Result of AbstractYarnClusterDescriptor#validateClusterResources is ignored
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10753">FLINK-10753</a>] - Propagate and log snapshotting exceptions
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10770">FLINK-10770</a>] - Some generated functions are not opened properly.
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10773">FLINK-10773</a>] - Resume externalized checkpoint end-to-end test fails
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10821">FLINK-10821</a>] - Resuming Externalized Checkpoint E2E test does not resume from Externalized Checkpoint
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10839">FLINK-10839</a>] - Fix implementation of PojoSerializer.duplicate() w.r.t. subclass serializer
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10856">FLINK-10856</a>] - Harden resume from externalized checkpoint E2E test
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10857">FLINK-10857</a>] - Conflict between JMX and Prometheus Metrics reporter
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10880">FLINK-10880</a>] - Failover strategies should not be applied to Batch Execution
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10913">FLINK-10913</a>] - ExecutionGraphRestartTest.testRestartAutomatically unstable on Travis
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10925">FLINK-10925</a>] - NPE in PythonPlanStreamer
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10990">FLINK-10990</a>] - Enforce minimum timespan in MeterView
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10998">FLINK-10998</a>] - flink-metrics-ganglia has LGPL dependency
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11011">FLINK-11011</a>] - Elasticsearch 6 sink end-to-end test unstable
+</li>
+</ul>
+
+<h2> Improvement
+</h2>
+<ul>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4173">FLINK-4173</a>] - Replace maven-assembly-plugin by maven-shade-plugin in flink-metrics
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-9869">FLINK-9869</a>] - Send PartitionInfo in batch to Improve perfornance
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10613">FLINK-10613</a>] - Remove logger casts in HBaseConnectorITCase
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10614">FLINK-10614</a>] - Update test_batch_allround.sh e2e to new testing infrastructure
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10637">FLINK-10637</a>] - Start MiniCluster with random REST port
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10678">FLINK-10678</a>] - Add a switch to run_test to configure if logs should be checked for errors/excepions
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10906">FLINK-10906</a>] - docker-entrypoint.sh logs credentails during startup
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10916">FLINK-10916</a>] - Include duplicated user-specified uid into error message
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11005">FLINK-11005</a>] - Define flink-sql-client uber-jar dependencies via artifactSet
+</li>
+</ul>
+
+<h2> Test
+</h2>
+<ul>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10606">FLINK-10606</a>] - Construct NetworkEnvironment simple for tests
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10607">FLINK-10607</a>] - Unify to remove duplicated NoOpResultPartitionConsumableNotifier
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10827">FLINK-10827</a>] - Add test for duplicate() to SerializerTestBase
+</li>
+</ul>
+</description>
+<pubDate>Wed, 26 Dec 2018 13:00:00 +0100</pubDate>
+<link>https://flink.apache.org/news/2018/12/26/release-1.5.6.html</link>
+<guid isPermaLink="true">/news/2018/12/26/release-1.5.6.html</guid>
+</item>
+
+<item>
+<title>Apache Flink 1.6.3 Released</title>
+<description><p>The Apache Flink community released the third bugfix version of the Apache Flink 1.6 series.</p>
+
+<p>This release includes more than 80 fixes and minor improvements for Flink 1.6.2. The list below includes a detailed list of all fixes.</p>
+
+<p>We highly recommend all users to upgrade to Flink 1.6.3.</p>
+
+<p>Updated Maven dependencies:</p>
+
+<div class="highlight"><pre><code class="language-xml"><span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-java<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.6.3<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span>
+<span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-streaming-java_2.11<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.6.3<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span>
+<span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-clients_2.11<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.6.3<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span></code></pre></div>
+
+<p>You can find the binaries on the updated <a href="http://flink.apache.org/downloads.html">Downloads page</a>.</p>
+
+<p>List of resolved issues:</p>
+
+<h2> Sub-task
+</h2>
+<ul>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10097">FLINK-10097</a>] - More tests to increase StreamingFileSink test coverage
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10252">FLINK-10252</a>] - Handle oversized metric messages
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10367">FLINK-10367</a>] - Avoid recursion stack overflow during releasing SingleInputGate
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10863">FLINK-10863</a>] - Assign uids to all operators in general purpose testing job
+</li>
+</ul>
+
+<h2> Bug
+</h2>
+<ul>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-8336">FLINK-8336</a>] - YarnFileStageTestS3ITCase.testRecursiveUploadForYarnS3 test instability
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-9635">FLINK-9635</a>] - Local recovery scheduling can cause spread out of tasks
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-9646">FLINK-9646</a>] - ExecutionGraphCoLocationRestartTest.testConstraintsAfterRestart failed on Travis
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-9878">FLINK-9878</a>] - IO worker threads BLOCKED on SSL Session Cache while CMS full gc
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10149">FLINK-10149</a>] - Fink Mesos allocates extra port when not configured to do so.
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10166">FLINK-10166</a>] - Dependency problems when executing SQL query in sql-client
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10309">FLINK-10309</a>] - Cancel with savepoint fails with java.net.ConnectException when using the per job-mode
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10357">FLINK-10357</a>] - Streaming File Sink end-to-end test failed with mismatch
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10359">FLINK-10359</a>] - Scala example in DataSet docs is broken
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10364">FLINK-10364</a>] - Test instability in NonHAQueryableStateFsBackendITCase#testMapState
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10419">FLINK-10419</a>] - ClassNotFoundException while deserializing user exceptions from checkpointing
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10425">FLINK-10425</a>] - taskmanager.host is not respected
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10455">FLINK-10455</a>] - Potential Kafka producer leak in case of failures
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10463">FLINK-10463</a>] - Null literal cannot be properly parsed in Java Table API function call
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10481">FLINK-10481</a>] - Wordcount end-to-end test in docker env unstable
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10482">FLINK-10482</a>] - java.lang.IllegalArgumentException: Negative number of in progress checkpoints
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10491">FLINK-10491</a>] - Deadlock during spilling data in SpillableSubpartition
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10566">FLINK-10566</a>] - Flink Planning is exponential in the number of stages
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10567">FLINK-10567</a>] - Lost serialize fields when ttl state store with the mutable serializer
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10570">FLINK-10570</a>] - State grows unbounded when &quot;within&quot; constraint not applied
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10581">FLINK-10581</a>] - YarnConfigurationITCase.testFlinkContainerMemory test instability
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10642">FLINK-10642</a>] - CodeGen split fields errors when maxGeneratedCodeLength equals 1
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10655">FLINK-10655</a>] - RemoteRpcInvocation not overwriting ObjectInputStream&#39;s ClassNotFoundException
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10663">FLINK-10663</a>] - Closing StreamingFileSink can cause NPE
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10669">FLINK-10669</a>] - Exceptions &amp; errors are not properly checked in logs in e2e tests
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10670">FLINK-10670</a>] - Fix Correlate codegen error
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10674">FLINK-10674</a>] - Fix handling of retractions after clean up
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10681">FLINK-10681</a>] - elasticsearch6.ElasticsearchSinkITCase fails if wrong JNA library installed
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10690">FLINK-10690</a>] - Tests leak resources via Files.list
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10693">FLINK-10693</a>] - Fix Scala EitherSerializer duplication
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10715">FLINK-10715</a>] - E2e tests fail with ConcurrentModificationException in MetricRegistryImpl
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10750">FLINK-10750</a>] - SocketClientSinkTest.testRetry fails on Travis
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10752">FLINK-10752</a>] - Result of AbstractYarnClusterDescriptor#validateClusterResources is ignored
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10753">FLINK-10753</a>] - Propagate and log snapshotting exceptions
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10763">FLINK-10763</a>] - Interval join produces wrong result type in Scala API
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10770">FLINK-10770</a>] - Some generated functions are not opened properly.
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10773">FLINK-10773</a>] - Resume externalized checkpoint end-to-end test fails
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10809">FLINK-10809</a>] - Using DataStreamUtils.reinterpretAsKeyedStream produces corrupted keyed state after restore
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10816">FLINK-10816</a>] - Fix LockableTypeSerializer.duplicate()
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10821">FLINK-10821</a>] - Resuming Externalized Checkpoint E2E test does not resume from Externalized Checkpoint
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10839">FLINK-10839</a>] - Fix implementation of PojoSerializer.duplicate() w.r.t. subclass serializer
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10842">FLINK-10842</a>] - Waiting loops are broken in e2e/common.sh
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10856">FLINK-10856</a>] - Harden resume from externalized checkpoint E2E test
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10857">FLINK-10857</a>] - Conflict between JMX and Prometheus Metrics reporter
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10880">FLINK-10880</a>] - Failover strategies should not be applied to Batch Execution
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10913">FLINK-10913</a>] - ExecutionGraphRestartTest.testRestartAutomatically unstable on Travis
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10925">FLINK-10925</a>] - NPE in PythonPlanStreamer
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10946">FLINK-10946</a>] - Resuming Externalized Checkpoint (rocks, incremental, scale up) end-to-end test failed on Travis
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10990">FLINK-10990</a>] - Enforce minimum timespan in MeterView
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10992">FLINK-10992</a>] - Jepsen: Do not use /tmp as HDFS Data Directory
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10997">FLINK-10997</a>] - Avro-confluent-registry does not bundle any dependency
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10998">FLINK-10998</a>] - flink-metrics-ganglia has LGPL dependency
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11011">FLINK-11011</a>] - Elasticsearch 6 sink end-to-end test unstable
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11017">FLINK-11017</a>] - Time interval for window aggregations in SQL is wrongly translated if specified with YEAR_MONTH resolution
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11029">FLINK-11029</a>] - Incorrect parameter in Working with state doc
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11041">FLINK-11041</a>] - ReinterpretDataStreamAsKeyedStreamITCase.testReinterpretAsKeyedStream failed on Travis
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11045">FLINK-11045</a>] - UserCodeClassLoader has not been set correctly for RuntimeUDFContext in CollectionExecutor
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11083">FLINK-11083</a>] - CRowSerializerConfigSnapshot is not instantiable
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11087">FLINK-11087</a>] - Broadcast state migration Incompatibility from 1.5.3 to 1.7.0
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11123">FLINK-11123</a>] - Missing import in ML quickstart docs
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11136">FLINK-11136</a>] - Fix the logical of merge for DISTINCT aggregates
+</li>
+</ul>
+
+<h2> Improvement
+</h2>
+<ul>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-4173">FLINK-4173</a>] - Replace maven-assembly-plugin by maven-shade-plugin in flink-metrics
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10353">FLINK-10353</a>] - Restoring a KafkaProducer with Semantic.EXACTLY_ONCE from a savepoint written with Semantic.AT_LEAST_ONCE fails with NPE
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10608">FLINK-10608</a>] - Add avro files generated by datastream-allround-test to RAT exclusions
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10613">FLINK-10613</a>] - Remove logger casts in HBaseConnectorITCase
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10614">FLINK-10614</a>] - Update test_batch_allround.sh e2e to new testing infrastructure
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10637">FLINK-10637</a>] - Start MiniCluster with random REST port
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10678">FLINK-10678</a>] - Add a switch to run_test to configure if logs should be checked for errors/excepions
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10692">FLINK-10692</a>] - Harden Confluent schema E2E test
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10883">FLINK-10883</a>] - Submitting a jobs without enough slots times out due to a unspecified timeout
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10906">FLINK-10906</a>] - docker-entrypoint.sh logs credentails during startup
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10916">FLINK-10916</a>] - Include duplicated user-specified uid into error message
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10951">FLINK-10951</a>] - Disable enforcing of YARN container virtual memory limits in tests
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11005">FLINK-11005</a>] - Define flink-sql-client uber-jar dependencies via artifactSet
+</li>
+</ul>
+
+<h2> Test
+</h2>
+<ul>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10606">FLINK-10606</a>] - Construct NetworkEnvironment simple for tests
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10607">FLINK-10607</a>] - Unify to remove duplicated NoOpResultPartitionConsumableNotifier
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10827">FLINK-10827</a>] - Add test for duplicate() to SerializerTestBase
+</li>
+</ul>
+
+<h2> Wish
+</h2>
+<ul>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10793">FLINK-10793</a>] - Change visibility of TtlValue and TtlSerializer to public for external tools
+</li>
+</ul>
+</description>
+<pubDate>Sat, 22 Dec 2018 13:00:00 +0100</pubDate>
+<link>https://flink.apache.org/news/2018/12/22/release-1.6.3.html</link>
+<guid isPermaLink="true">/news/2018/12/22/release-1.6.3.html</guid>
+</item>
+
+<item>
+<title>Apache Flink 1.7.1 Released</title>
+<description><p>The Apache Flink community released the first bugfix version of the Apache Flink 1.7 series.</p>
+
+<p>This release includes 27 fixes and minor improvements for Flink 1.7.0. The list below includes a detailed list of all fixes.</p>
+
+<p>We highly recommend all users to upgrade to Flink 1.7.1.</p>
+
+<p>Updated Maven dependencies:</p>
+
+<div class="highlight"><pre><code class="language-xml"><span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-java<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.7.1<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span>
+<span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-streaming-java_2.11<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.7.1<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span>
+<span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-clients_2.11<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.7.1<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span></code></pre></div>
+
+<p>You can find the binaries on the updated <a href="http://flink.apache.org/downloads.html">Downloads page</a>.</p>
+
+<p>List of resolved issues:</p>
+
+<h2> Sub-task
+</h2>
+<ul>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10252">FLINK-10252</a>] - Handle oversized metric messages
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10367">FLINK-10367</a>] - Avoid recursion stack overflow during releasing SingleInputGate
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10522">FLINK-10522</a>] - Check if RecoverableWriter supportsResume and act accordingly.
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10963">FLINK-10963</a>] - Cleanup small objects uploaded to S3 as independent objects
+</li>
+</ul>
+
+<h2> Bug
+</h2>
+<ul>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-8336">FLINK-8336</a>] - YarnFileStageTestS3ITCase.testRecursiveUploadForYarnS3 test instability
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-9646">FLINK-9646</a>] - ExecutionGraphCoLocationRestartTest.testConstraintsAfterRestart failed on Travis
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10149">FLINK-10149</a>] - Fink Mesos allocates extra port when not configured to do so.
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10359">FLINK-10359</a>] - Scala example in DataSet docs is broken
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10482">FLINK-10482</a>] - java.lang.IllegalArgumentException: Negative number of in progress checkpoints
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10566">FLINK-10566</a>] - Flink Planning is exponential in the number of stages
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10997">FLINK-10997</a>] - Avro-confluent-registry does not bundle any dependency
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11011">FLINK-11011</a>] - Elasticsearch 6 sink end-to-end test unstable
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11013">FLINK-11013</a>] - Fix distinct aggregates for group window in Table API
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11017">FLINK-11017</a>] - Time interval for window aggregations in SQL is wrongly translated if specified with YEAR_MONTH resolution
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11029">FLINK-11029</a>] - Incorrect parameter in Working with state doc
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11032">FLINK-11032</a>] - Elasticsearch (v6.3.1) sink end-to-end test unstable on Travis
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11033">FLINK-11033</a>] - Elasticsearch (v6.3.1) sink end-to-end test unstable on Travis
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11041">FLINK-11041</a>] - ReinterpretDataStreamAsKeyedStreamITCase.testReinterpretAsKeyedStream failed on Travis
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11044">FLINK-11044</a>] - RegisterTableSink docs incorrect
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11045">FLINK-11045</a>] - UserCodeClassLoader has not been set correctly for RuntimeUDFContext in CollectionExecutor
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11047">FLINK-11047</a>] - CoGroupGroupSortTranslationTest does not compile with scala 2.12
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11085">FLINK-11085</a>] - NoClassDefFoundError in presto-s3 filesystem
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11087">FLINK-11087</a>] - Broadcast state migration Incompatibility from 1.5.3 to 1.7.0
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11094">FLINK-11094</a>] - Restored state in RocksDBStateBackend that has not been accessed in restored execution causes NPE on snapshot
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11123">FLINK-11123</a>] - Missing import in ML quickstart docs
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11136">FLINK-11136</a>] - Fix the logical of merge for DISTINCT aggregates
+</li>
+</ul>
+
+<h2> Improvement
+</h2>
+<ul>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-11080">FLINK-11080</a>] - Define flink-connector-elasticsearch6 uber-jar dependencies via artifactSet
+</li>
+</ul>
+</description>
+<pubDate>Fri, 21 Dec 2018 13:00:00 +0100</pubDate>
+<link>https://flink.apache.org/news/2018/12/21/release-1.7.1.html</link>
+<guid isPermaLink="true">/news/2018/12/21/release-1.7.1.html</guid>
+</item>
+
+<item>
+<title>Apache Flink 1.7.0 Release Announcement</title>
+<description><p>The Apache Flink community is pleased to announce Apache Flink 1.7.0.
+The latest release includes more than 420 resolved issues and some exciting additions to Flink that we describe in the following sections of this post.
+Please check the <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&amp;version=12343585">complete changelog</a> for more details.</p>
+
+<p>Flink 1.7.0 is API-compatible with previous 1.x.y releases for APIs annotated with the <code>@Public</code> annotation.
+The release is available now and we encourage everyone to <a href="http://flink.apache.org/downloads.html">download the release</a> and check out the updated <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.7/">documentation</a>.
+Feedback through the Flink <a href="http://flink.apache.org/community.html#mailing-lists">mailing lists</a> or <a href="https://issues.apache.org/jira/projects/FLINK/summary">JIRA</a> is, as always, very much appreciated!</p>
+
+<p>You can find the binaries on the updated <a href="http://flink.apache.org/downloads.html">Downloads page</a> on the Flink project site.</p>
+
+<div class="page-toc">
+<ul id="markdown-toc">
+ <li><a href="#flink-170---extending-the-reach-of-stream-processing" id="markdown-toc-flink-170---extending-the-reach-of-stream-processing">Flink 1.7.0 - Extending the reach of Stream Processing</a></li>
+ <li><a href="#new-features-and-improvements" id="markdown-toc-new-features-and-improvements">New Features and Improvements</a></li>
+ <li><a href="#release-notes" id="markdown-toc-release-notes">Release Notes</a></li>
+ <li><a href="#list-of-contributors" id="markdown-toc-list-of-contributors">List of Contributors</a></li>
+</ul>
+
+</div>
+
+<h2 id="flink-170---extending-the-reach-of-stream-processing">Flink 1.7.0 - Extending the reach of Stream Processing</h2>
+
+<p>In Flink 1.7.0 we come closer to our goals of enabling fast data processing and building data-intensive applications for the Flink community in a seamless way.
+Our latest release includes some exciting new features and improvements such as support for Scala 2.12, an exactly-once S3 file sink, the integration of complex event processing with streaming SQL and more features that we explain below.</p>
+
+<h2 id="new-features-and-improvements">New Features and Improvements</h2>
+
+<ul>
+ <li>
+ <p><strong>Scala 2.12 Support in Apache Flink</strong> (<a href="https://issues.apache.org/jira/browse/FLINK-7811">FLINK-7811</a>):
+Apache Flink 1.7.0 is the first release which comes with full support for Scala 2.12.
+This allows users to write Flink applications with a newer Scala version and to leverage the Scala 2.12 ecosystem.</p>
+ </li>
+ <li>
+ <p><strong>State Evolution</strong> (<a href="https://issues.apache.org/jira/browse/FLINK-9376">FLINK-9376</a>):
+In many cases, a long-running Flink application needs to evolve during its lifetime because of changing requirements.
+Changing the user state without losing the current application progress in the form of its state is a crucial requirement for application evolution.</p>
+
+ <p>With Flink 1.7.0, the community added state evolution which allows you to flexibly adapt a long-running application’s user states schema, while maintaining compatibility with previous savepoints.
+With state evolution it is possible to add or remove columns to your state schema in order to change which business features will be captured by your application after it has been deployed.</p>
+
+ <p>State schema evolution now works out-of-the-box when using Avro’s generated classes as user state, meaning that the schema of the state can be evolved according to Avro’s specifications.
+While Avro types are the only built-in type that supports schema evolution as of Flink 1.7, the community continues working to further extend support to other types in future Flink releases.</p>
+ </li>
+ <li>
+ <p><strong>Exactly-once S3 StreamingFileSink</strong> (<a href="https://issues.apache.org/jira/browse/FLINK-9752">FLINK-9752</a>):
+The <code>StreamingFileSink</code> which was introduced in Flink 1.6.0 is now extended to also support writing to S3 filesystems with exactly-once processing guarantees.
+Using this feature allows users to build exactly-once end-to-end pipelines writing to S3.</p>
+ </li>
+ <li>
+ <p><strong><code>MATCH_RECOGNIZE</code> Support in Streaming SQL</strong> (<a href="https://issues.apache.org/jira/browse/FLINK-6935">FLINK-6935</a>):
+This is a major addition to Apache Flink 1.7.0 that provides initial support of the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.7/dev/table/streaming/match_recognize.html"><code>MATCH_RECOGNIZE</code></a> standard to Flink SQL.
+This feature combines both complex event processing (CEP) and SQL for easy pattern matching on data streams and, thus, enabling a whole set of new use cases.</p>
+
+ <p>This feature is currently in beta phase so we welcome any feedback and suggestions from the community for future iterations and improvements.</p>
+ </li>
+ <li>
+ <p><strong>Temporal Tables and Temporal Joins in Streaming SQL</strong> (<a href="https://issues.apache.org/jira/browse/FLINK-9712">FLINK-9712</a>):
+Temporal Tables is a new concept in Apache Flink that gives a (parameterized) view on a table’s changing history and returns the content of a table at a specific point in time.</p>
+
+ <p>As an example, we can use a table with historical currency exchange rates.
+Such a table is constantly growing/evolving as time progresses and newly updated exchange rates are added.
+Temporal Table is a view that can return the actual state of those exchange rates to any given point of time.
+With such a table it is possible to convert a stream of orders in different currencies to a common currency using the correct exchange rate.</p>
+
+ <p>Temporal Joins allow for memory and computational-efficient joins of Streaming data with an ever-changing/updating table, using either processing time or event time, while being ANSI SQL compliant.</p>
+ </li>
+ <li>
+ <p><strong>Miscellaneous Features for Streaming SQL</strong>:
+Besides the major features mentioned above, Flink’s Table &amp; SQL API has been extended to serve more use cases.</p>
+
+ <p>The following built-in functions were added to the APIs: <code>TO_BASE64</code>, <code>LOG2</code>, <code>LTRIM</code>, <code>REPEAT</code>, <code>REPLACE</code>, <code>COSH</code>, <code>SINH</code>, <code>TANH</code></p>
+
+ <p>The SQL Client now supports the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.7/dev/table/sqlClient.html#sql-views">definition of views</a> both in an environment file and within a CLI session.
+Furthermore, basic SQL statement auto-completion has been added to the CLI.</p>
+
+ <p>The community added an <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.7/dev/table/connect.html#elasticsearch-connector">Elasticsearch 6 table sink</a> which allows to store updating results of a dynamic table.</p>
+ </li>
+ <li>
+ <p><strong>Versioned REST API</strong> (<a href="https://issues.apache.org/jira/browse/FLINK-7551">FLINK-7551</a>):
+Beginning with Flink 1.7.0, the REST API is versioned.
+This guarantees the stability of Flink’s REST API so that third-party applications can be developed against a stable API in Flink.
+Thus, future Flink upgrades will not require changes to existing third-party integrations.</p>
+ </li>
+ <li>
+ <p><strong>Kafka 2.0 Connector</strong> (<a href="https://issues.apache.org/jira/browse/FLINK-10598">FLINK-10598</a>):
+Apache Flink 1.7.0 continues to add more connectors, making it even easier to interact with more external systems.
+In this release, the community added the Kafka 2.0 connector which allows to read from and write to Kafka 2.0 with exactly-once guarantees.</p>
+ </li>
+ <li>
+ <p><strong>Local Recovery</strong> (<a href="https://issues.apache.org/jira/browse/FLINK-9635">FLINK-9635</a>):
+Apache Flink 1.7.0 completes the local recovery feature by extending Flink’s scheduling to take previous deployment locations into account in case of recovery.</p>
+
+ <p>If local recovery is enabled Flink will keep a local copy of the latest checkpoint on the machine where the task is running.
+By scheduling tasks to their previous locations, Flink will, thus, minimize the network traffic for restoring state by reading checkpoint state from local disk.
+This feature considerably improves recovery speed.</p>
+ </li>
+ <li>
+ <p><strong>Removal of Flink’s Legacy Mode</strong> (<a href="https://issues.apache.org/jira/browse/FLINK-10392">FLINK-10392</a>):
+Apache Flink 1.7.0 marks the release where the Flip-6 effort has been fully completed and reached feature parity with the legacy mode.
+Consequently, this release removes support for the legacy mode.</p>
+ </li>
+</ul>
+
+<h2 id="release-notes">Release Notes</h2>
+
+<p>Please review the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.7/release-notes/flink-1.7.html">release notes</a> if you plan to upgrade your Flink setup to Flink 1.7.</p>
+
+<h2 id="list-of-contributors">List of Contributors</h2>
+
+<p>We would like to acknowledge all community members for contributing to this release.
+Special credits go to the following members for contributing to the 1.7.0 release (according to git):</p>
+
+<p>Aitozi, Alex Arkhipov, Alexander Koltsov, Alexey Trenikhin, Alice, Alice Yan, Aljoscha Krettek, Andrei Poluliakh, Andrey Zagrebin, Ashwin Sinha, Barisa Obradovic, Ben La Monica, Benoit Meriaux, Bowen Li, Chesnay Schepler, Christophe Jolif, Congxian Qiu, Craig Foster, David Anderson, Dawid Wysakowicz, Dian Fu, Diego Carvallo, Dimitris Palyvos, Eugen Yushin, Fabian Hueske, Florian Schmidt, Gary Yao, Guibo Pan, Hequn Cheng, Hiroaki Yoshida, Igal Shilman, JIN SUN, Jamie Grier, Jayan [...]
+</description>
+<pubDate>Fri, 30 Nov 2018 13:00:00 +0100</pubDate>
+<link>https://flink.apache.org/news/2018/11/30/release-1.7.0.html</link>
+<guid isPermaLink="true">/news/2018/11/30/release-1.7.0.html</guid>
+</item>
+
+<item>
+<title>Apache Flink 1.6.2 Released</title>
+<description><p>The Apache Flink community released the second bugfix version of the Apache Flink 1.6 series.</p>
+
+<p>This release includes more than 30 fixes and minor improvements for Flink 1.6.1. The list below includes a detailed list of all fixes.</p>
+
+<p>We highly recommend all users to upgrade to Flink 1.6.2.</p>
+
+<p>Updated Maven dependencies:</p>
+
+<div class="highlight"><pre><code class="language-xml"><span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-java<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.6.2<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span>
+<span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-streaming-java_2.11<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.6.2<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span>
+<span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-clients_2.11<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.6.2<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span></code></pre></div>
+
+<p>You can find the binaries on the updated <a href="http://flink.apache.org/downloads.html">Downloads page</a>.</p>
+
+<p>List of resolved issues:</p>
+
+<h2> Sub-task
+</h2>
+<ul>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10242">FLINK-10242</a>] - Latency marker interval should be configurable
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10243">FLINK-10243</a>] - Add option to reduce latency metrics granularity
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10331">FLINK-10331</a>] - Reduce number of flush requests to the network stack
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10332">FLINK-10332</a>] - Move data available notification in PipelinedSubpartition out of the synchronized block
+</li>
+</ul>
+
+<h2> Bug
+</h2>
+<ul>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5542">FLINK-5542</a>] - YARN client incorrectly uses local YARN config to check vcore capacity
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-9567">FLINK-9567</a>] - Flink does not release resource in Yarn Cluster mode
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-9788">FLINK-9788</a>] - ExecutionGraph Inconsistency prevents Job from recovering
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-9884">FLINK-9884</a>] - Slot request may not be removed when it has already be assigned in slot manager
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-9891">FLINK-9891</a>] - Flink cluster is not shutdown in YARN mode when Flink client is stopped
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-9932">FLINK-9932</a>] - Timed-out TaskExecutor slot-offers to JobMaster leak the slot
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10135">FLINK-10135</a>] - Certain cluster-level metrics are no longer exposed
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10157">FLINK-10157</a>] - Allow `null` user values in map state with TTL
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10222">FLINK-10222</a>] - Table scalar function expression parses error when function name equals the exists keyword suffix
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10259">FLINK-10259</a>] - Key validation for GroupWindowAggregate is broken
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10263">FLINK-10263</a>] - User-defined function with LITERAL paramters yields CompileException
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10316">FLINK-10316</a>] - Add check to KinesisProducer that aws.region is set
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10354">FLINK-10354</a>] - Savepoints should be counted as retained checkpoints
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10363">FLINK-10363</a>] - S3 FileSystem factory prints secrets into logs
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10379">FLINK-10379</a>] - Can not use Table Functions in Java Table API
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10383">FLINK-10383</a>] - Hadoop configurations on the classpath seep into the S3 file system configs
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10390">FLINK-10390</a>] - DataDog MetricReporter leaks connections
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10400">FLINK-10400</a>] - Return failed JobResult if job terminates in state FAILED or CANCELED
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10415">FLINK-10415</a>] - RestClient does not react to lost connection
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10444">FLINK-10444</a>] - Make S3 entropy injection work with FileSystem safety net
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10451">FLINK-10451</a>] - TableFunctionCollector should handle the life cycle of ScalarFunction
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10465">FLINK-10465</a>] - Jepsen: runit supervised sshd is stopped on tear down
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10469">FLINK-10469</a>] - FileChannel may not write the whole buffer in a single call to FileChannel.write(Buffer buffer)
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10487">FLINK-10487</a>] - fix invalid Flink SQL example
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10516">FLINK-10516</a>] - YarnApplicationMasterRunner does not initialize FileSystem with correct Flink Configuration during setup
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10524">FLINK-10524</a>] - MemoryManagerConcurrentModReleaseTest.testConcurrentModificationWhileReleasing failed on travis
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10532">FLINK-10532</a>] - Broken links in documentation
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10544">FLINK-10544</a>] - Remove custom settings.xml for snapshot deployments
+</li>
+</ul>
+
+<h2> Improvement
+</h2>
+<ul>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-9061">FLINK-9061</a>] - Add entropy to s3 path for better scalability
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10075">FLINK-10075</a>] - HTTP connections to a secured REST endpoint flood the log
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10260">FLINK-10260</a>] - Confusing log messages during TaskManager registration
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10282">FLINK-10282</a>] - Provide separate thread-pool for REST endpoint
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10291">FLINK-10291</a>] - Generate JobGraph with fixed/configurable JobID in StandaloneJobClusterEntrypoint
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10311">FLINK-10311</a>] - HA end-to-end/Jepsen tests for standby Dispatchers
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10312">FLINK-10312</a>] - Wrong / missing exception when submitting job
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10371">FLINK-10371</a>] - Allow to enable SSL mutual authentication on REST endpoints by configuration
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10375">FLINK-10375</a>] - ExceptionInChainedStubException hides wrapped exception in cause
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10582">FLINK-10582</a>] - Make REST executor thread priority configurable
+</li>
+</ul>
+</description>
+<pubDate>Mon, 29 Oct 2018 13:00:00 +0100</pubDate>
+<link>https://flink.apache.org/news/2018/10/29/release-1.6.2.html</link>
+<guid isPermaLink="true">/news/2018/10/29/release-1.6.2.html</guid>
+</item>
+
+<item>
+<title>Apache Flink 1.5.5 Released</title>
+<description><p>The Apache Flink community released the fifth bugfix version of the Apache Flink 1.5 series.</p>
+
+<p>This release includes more than 20 fixes and minor improvements for Flink 1.5.4. The list below includes a detailed list of all fixes.</p>
+
+<p>We highly recommend all users to upgrade to Flink 1.5.5.</p>
+
+<p>Updated Maven dependencies:</p>
+
+<div class="highlight"><pre><code class="language-xml"><span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-java<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.5.5<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span>
+<span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-streaming-java_2.11<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.5.5<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span>
+<span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-clients_2.11<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.5.5<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span></code></pre></div>
+
+<p>You can find the binaries on the updated <a href="http://flink.apache.org/downloads.html">Downloads page</a>.</p>
+
+<p>List of resolved issues:</p>
+
+<h2> Sub-task
+</h2>
+<ul>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10242">FLINK-10242</a>] - Latency marker interval should be configurable
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10243">FLINK-10243</a>] - Add option to reduce latency metrics granularity
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10331">FLINK-10331</a>] - Reduce number of flush requests to the network stack
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10332">FLINK-10332</a>] - Move data available notification in PipelinedSubpartition out of the synchronized block
+</li>
+</ul>
+
+<h2> Bug
+</h2>
+<ul>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5542">FLINK-5542</a>] - YARN client incorrectly uses local YARN config to check vcore capacity
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-9567">FLINK-9567</a>] - Flink does not release resource in Yarn Cluster mode
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-9788">FLINK-9788</a>] - ExecutionGraph Inconsistency prevents Job from recovering
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-9884">FLINK-9884</a>] - Slot request may not be removed when it has already be assigned in slot manager
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-9891">FLINK-9891</a>] - Flink cluster is not shutdown in YARN mode when Flink client is stopped
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-9932">FLINK-9932</a>] - Timed-out TaskExecutor slot-offers to JobMaster leak the slot
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10135">FLINK-10135</a>] - Certain cluster-level metrics are no longer exposed
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10222">FLINK-10222</a>] - Table scalar function expression parses error when function name equals the exists keyword suffix
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10259">FLINK-10259</a>] - Key validation for GroupWindowAggregate is broken
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10316">FLINK-10316</a>] - Add check to KinesisProducer that aws.region is set
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10354">FLINK-10354</a>] - Savepoints should be counted as retained checkpoints
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10400">FLINK-10400</a>] - Return failed JobResult if job terminates in state FAILED or CANCELED
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10415">FLINK-10415</a>] - RestClient does not react to lost connection
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10451">FLINK-10451</a>] - TableFunctionCollector should handle the life cycle of ScalarFunction
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10469">FLINK-10469</a>] - FileChannel may not write the whole buffer in a single call to FileChannel.write(Buffer buffer)
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10487">FLINK-10487</a>] - fix invalid Flink SQL example
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10516">FLINK-10516</a>] - YarnApplicationMasterRunner does not initialize FileSystem with correct Flink Configuration during setup
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10524">FLINK-10524</a>] - MemoryManagerConcurrentModReleaseTest.testConcurrentModificationWhileReleasing failed on travis
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10544">FLINK-10544</a>] - Remove custom settings.xml for snapshot deployments
+</li>
+</ul>
+
+<h2> Improvement
+</h2>
+<ul>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10075">FLINK-10075</a>] - HTTP connections to a secured REST endpoint flood the log
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10260">FLINK-10260</a>] - Confusing log messages during TaskManager registration
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10282">FLINK-10282</a>] - Provide separate thread-pool for REST endpoint
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10312">FLINK-10312</a>] - Wrong / missing exception when submitting job
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10375">FLINK-10375</a>] - ExceptionInChainedStubException hides wrapped exception in cause
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10582">FLINK-10582</a>] - Make REST executor thread priority configurable
+</li>
+</ul>
+</description>
+<pubDate>Mon, 29 Oct 2018 13:00:00 +0100</pubDate>
+<link>https://flink.apache.org/news/2018/10/29/release-1.5.5.html</link>
+<guid isPermaLink="true">/news/2018/10/29/release-1.5.5.html</guid>
+</item>
+
+<item>
+<title>Apache Flink 1.6.1 Released</title>
+<description><p>The Apache Flink community released the first bugfix version of the Apache Flink 1.6 series.</p>
+
+<p>This release includes 60 fixes and minor improvements for Flink 1.6.1. The list below includes a detailed list of all fixes.</p>
+
+<p>We highly recommend all users to upgrade to Flink 1.6.1.</p>
+
+<p>Updated Maven dependencies:</p>
+
+<div class="highlight"><pre><code class="language-xml"><span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-java<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.6.1<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span>
+<span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-streaming-java_2.11<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.6.1<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span>
+<span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-clients_2.11<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.6.1<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span></code></pre></div>
+
+<p>You can find the binaries on the updated <a href="http://flink.apache.org/downloads.html">Downloads page</a>.</p>
+
+<p>List of resolved issues:</p>
+
+<h2> Sub-task
+</h2>
+<ul>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-9637">FLINK-9637</a>] - Add public user documentation for TTL feature
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10068">FLINK-10068</a>] - Add documentation for async/RocksDB-based timers
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10085">FLINK-10085</a>] - Update AbstractOperatorRestoreTestBase
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10087">FLINK-10087</a>] - Update BucketingSinkMigrationTest
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10089">FLINK-10089</a>] - Update FlinkKafkaConsumerBaseMigrationTest
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10090">FLINK-10090</a>] - Update ContinuousFileProcessingMigrationTest
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10091">FLINK-10091</a>] - Update WindowOperatorMigrationTest
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10092">FLINK-10092</a>] - Update StatefulJobSavepointMigrationITCase
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10109">FLINK-10109</a>] - Add documentation for StreamingFileSink
+</li>
+</ul>
+
+<h2> Bug
+</h2>
+<ul>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-9289">FLINK-9289</a>] - Parallelism of generated operators should have max parallism of input
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-9546">FLINK-9546</a>] - The heartbeatTimeoutIntervalMs of HeartbeatMonitor should be larger than 0
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-9693">FLINK-9693</a>] - Possible memory leak in jobmanager retaining archived checkpoints
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-9972">FLINK-9972</a>] - Debug memory logging not working
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10011">FLINK-10011</a>] - Old job resurrected during HA failover
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10063">FLINK-10063</a>] - Jepsen: Automatically restart Mesos Processes
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10101">FLINK-10101</a>] - Mesos web ui url is missing.
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10105">FLINK-10105</a>] - Test failure because of jobmanager.execution.failover-strategy is outdated
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10115">FLINK-10115</a>] - Content-length limit is also applied to FileUploads
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10116">FLINK-10116</a>] - createComparator fails on case class with Unit type fields prior to the join-key
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10141">FLINK-10141</a>] - Reduce lock contention introduced with 1.5
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10142">FLINK-10142</a>] - Reduce synchronization overhead for credit notifications
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10150">FLINK-10150</a>] - Chained batch operators interfere with each other other
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10151">FLINK-10151</a>] - [State TTL] Fix false recursion call in TransformingStateTableKeyGroupPartitioner.tryAddToSource
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10154">FLINK-10154</a>] - Make sure we always read at least one record in KinesisConnector
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10169">FLINK-10169</a>] - RowtimeValidator fails with custom TimestampExtractor
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10172">FLINK-10172</a>] - Inconsistentcy in ExpressionParser and ExpressionDsl for order by asc/desc
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10192">FLINK-10192</a>] - SQL Client table visualization mode does not update correctly
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10193">FLINK-10193</a>] - Default RPC timeout is used when triggering savepoint via JobMasterGateway
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10204">FLINK-10204</a>] - StreamElementSerializer#copy broken for LatencyMarkers
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10255">FLINK-10255</a>] - Standby Dispatcher locks submitted JobGraphs
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10261">FLINK-10261</a>] - INSERT INTO does not work with ORDER BY clause
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10267">FLINK-10267</a>] - [State] Fix arbitrary iterator access on RocksDBMapIterator
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10269">FLINK-10269</a>] - Elasticsearch 6 UpdateRequest fail because of binary incompatibility
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10283">FLINK-10283</a>] - FileCache logs unnecessary warnings
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10293">FLINK-10293</a>] - RemoteStreamEnvironment does not forward port to RestClusterClient
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10314">FLINK-10314</a>] - Blocking calls in Execution Graph creation bring down cluster
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10328">FLINK-10328</a>] - Stopping the ZooKeeperSubmittedJobGraphStore should release all currently held locks
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10329">FLINK-10329</a>] - Fail with exception if job cannot be removed by ZooKeeperSubmittedJobGraphStore#removeJobGraph
+</li>
+</ul>
+
+<h2> New Feature
+</h2>
+<ul>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10022">FLINK-10022</a>] - Add metrics for input/output buffers
+</li>
+</ul>
+
+<h2> Improvement
+</h2>
+<ul>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-9013">FLINK-9013</a>] - Document yarn.containers.vcores only being effective when adapting YARN config
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-9446">FLINK-9446</a>] - Compatibility table not up-to-date
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-9795">FLINK-9795</a>] - Update Mesos documentation for flip6
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-9859">FLINK-9859</a>] - More Akka config options
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-9899">FLINK-9899</a>] - Add more metrics to the Kinesis source connector
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-9962">FLINK-9962</a>] - allow users to specify TimeZone in DateTimeBucketer
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10001">FLINK-10001</a>] - Improve Kubernetes documentation
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10006">FLINK-10006</a>] - Improve logging in BarrierBuffer
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10020">FLINK-10020</a>] - Kinesis Consumer listShards should support more recoverable exceptions
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10082">FLINK-10082</a>] - Initialize StringBuilder in Slf4jReporter with estimated size
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10094">FLINK-10094</a>] - Always backup default config for end-to-end tests
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10110">FLINK-10110</a>] - Harden e2e Kafka shutdown
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10131">FLINK-10131</a>] - Improve logging around ResultSubpartition
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10137">FLINK-10137</a>] - YARN: Log completed Containers
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10164">FLINK-10164</a>] - Add support for resuming from savepoints to StandaloneJobClusterEntrypoint
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10170">FLINK-10170</a>] - Support string representation for map and array types in descriptor-based Table API
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10185">FLINK-10185</a>] - Make ZooKeeperStateHandleStore#releaseAndTryRemove synchronous
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10223">FLINK-10223</a>] - TaskManagers should log their ResourceID during startup
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10301">FLINK-10301</a>] - Allow a custom Configuration in StreamNetworkBenchmarkEnvironment
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10325">FLINK-10325</a>] - [State TTL] Refactor TtlListState to use only loops, no java stream API for performance
+</li>
+</ul>
+
+<h2> Test
+</h2>
+<ul>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10084">FLINK-10084</a>] - Migration tests weren&#39;t updated for 1.5
+</li>
+</ul>
+</description>
+<pubDate>Thu, 20 Sep 2018 14:00:00 +0200</pubDate>
+<link>https://flink.apache.org/news/2018/09/20/release-1.6.1.html</link>
+<guid isPermaLink="true">/news/2018/09/20/release-1.6.1.html</guid>
+</item>
+
+<item>
+<title>Apache Flink 1.5.4 Released</title>
+<description><p>The Apache Flink community released the fourth bugfix version of the Apache Flink 1.5 series.</p>
+
+<p>This release includes more than 20 fixes and minor improvements for Flink 1.5.4. The list below includes a detailed list of all fixes.</p>
+
+<p>We highly recommend all users to upgrade to Flink 1.5.4.</p>
+
+<p>Updated Maven dependencies:</p>
+
+<div class="highlight"><pre><code class="language-xml"><span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-java<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.5.4<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span>
+<span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-streaming-java_2.11<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.5.4<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span>
+<span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-clients_2.11<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.5.4<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span></code></pre></div>
+
+<p>You can find the binaries on the updated <a href="http://flink.apache.org/downloads.html">Downloads page</a>.</p>
+
+<p>List of resolved issues:</p>
+
+<h2> Bug
+</h2>
+<ul>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-9878">FLINK-9878</a>] - IO worker threads BLOCKED on SSL Session Cache while CMS full gc
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10011">FLINK-10011</a>] - Old job resurrected during HA failover
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10101">FLINK-10101</a>] - Mesos web ui url is missing.
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10115">FLINK-10115</a>] - Content-length limit is also applied to FileUploads
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10116">FLINK-10116</a>] - createComparator fails on case class with Unit type fields prior to the join-key
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10141">FLINK-10141</a>] - Reduce lock contention introduced with 1.5
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10142">FLINK-10142</a>] - Reduce synchronization overhead for credit notifications
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10150">FLINK-10150</a>] - Chained batch operators interfere with each other other
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10172">FLINK-10172</a>] - Inconsistentcy in ExpressionParser and ExpressionDsl for order by asc/desc
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10193">FLINK-10193</a>] - Default RPC timeout is used when triggering savepoint via JobMasterGateway
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10204">FLINK-10204</a>] - StreamElementSerializer#copy broken for LatencyMarkers
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10255">FLINK-10255</a>] - Standby Dispatcher locks submitted JobGraphs
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10261">FLINK-10261</a>] - INSERT INTO does not work with ORDER BY clause
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10267">FLINK-10267</a>] - [State] Fix arbitrary iterator access on RocksDBMapIterator
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10293">FLINK-10293</a>] - RemoteStreamEnvironment does not forward port to RestClusterClient
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10314">FLINK-10314</a>] - Blocking calls in Execution Graph creation bring down cluster
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10328">FLINK-10328</a>] - Stopping the ZooKeeperSubmittedJobGraphStore should release all currently held locks
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10329">FLINK-10329</a>] - Fail with exception if job cannot be removed by ZooKeeperSubmittedJobGraphStore#removeJobGraph
+</li>
+</ul>
+
+<h2> Improvement
+</h2>
+<ul>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10082">FLINK-10082</a>] - Initialize StringBuilder in Slf4jReporter with estimated size
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10131">FLINK-10131</a>] - Improve logging around ResultSubpartition
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10137">FLINK-10137</a>] - YARN: Log completed Containers
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10185">FLINK-10185</a>] - Make ZooKeeperStateHandleStore#releaseAndTryRemove synchronous
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10223">FLINK-10223</a>] - TaskManagers should log their ResourceID during startup
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-10301">FLINK-10301</a>] - Allow a custom Configuration in StreamNetworkBenchmarkEnvironment
+</li>
+</ul>
+</description>
+<pubDate>Thu, 20 Sep 2018 14:00:00 +0200</pubDate>
+<link>https://flink.apache.org/news/2018/09/20/release-1.5.4.html</link>
+<guid isPermaLink="true">/news/2018/09/20/release-1.5.4.html</guid>
+</item>
+
+<item>
+<title>Apache Flink 1.5.3 Released</title>
+<description><p>The Apache Flink community released the third bugfix version of the Apache Flink 1.5 series.</p>
+
+<p>This release includes more than 20 fixes and minor improvements for Flink 1.5.3. The list below includes a detailed list of all fixes.</p>
+
+<p>We highly recommend all users to upgrade to Flink 1.5.3.</p>
+
+<p>Updated Maven dependencies:</p>
+
+<div class="highlight"><pre><code class="language-xml"><span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-java<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.5.3<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span>
+<span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-streaming-java_2.11<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.5.3<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span>
+<span class="nt">&lt;dependency&gt;</span>
+ <span class="nt">&lt;groupId&gt;</span>org.apache.flink<span class="nt">&lt;/groupId&gt;</span>
+ <span class="nt">&lt;artifactId&gt;</span>flink-clients_2.11<span class="nt">&lt;/artifactId&gt;</span>
+ <span class="nt">&lt;version&gt;</span>1.5.3<span class="nt">&lt;/version&gt;</span>
+<span class="nt">&lt;/dependency&gt;</span></code></pre></div>
+
+<p>You can find the binaries on the updated <a href="http://flink.apache.org/downloads.html">Downloads page</a>.</p>
+
+<p>List of resolved issues:</p>
+
+<h2> Sub-task
+</h2>
+<ul>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-9951">FLINK-9951</a>] - Update scm developerConnection
+</li>
+</ul>
+
+<h2> Bug
+</h2>
+<ul>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-5750">FLINK-5750</a>] - Incorrect translation of n-ary Union
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-9289">FLINK-9289</a>] - Parallelism of generated operators should have max parallism of input
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-9546">FLINK-9546</a>] - The heartbeatTimeoutIntervalMs of HeartbeatMonitor should be larger than 0
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-9655">FLINK-9655</a>] - Externalized checkpoint E2E test fails on travis
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-9693">FLINK-9693</a>] - Possible memory leak in jobmanager retaining archived checkpoints
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-9694">FLINK-9694</a>] - Potentially NPE in CompositeTypeSerializerConfigSnapshot constructor
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-9923">FLINK-9923</a>] - OneInputStreamTaskTest.testWatermarkMetrics fails on Travis
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-9935">FLINK-9935</a>] - Batch Table API: grouping by window and attribute causes java.lang.ClassCastException:
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-9936">FLINK-9936</a>] - Mesos resource manager unable to connect to master after failover
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-9946">FLINK-9946</a>] - Quickstart E2E test archetype version is hard-coded
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-9969">FLINK-9969</a>] - Unreasonable memory requirements to complete examples/batch/WordCount
+</li>
+<li>[<a href="https://issues.apache.org/jira/browse/FLINK-9972">FLINK-9972</a>] - Debug memory logging not working
+</li>
... 7397 lines suppressed ...