You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by fh...@apache.org on 2019/09/13 12:18:48 UTC
[flink-web] 03/03: Rebuild website
This is an automated email from the ASF dual-hosted git repository.
fhueske pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/flink-web.git
commit 3003e0ec109f5ba0a6670d5dd0e107265b1c7638
Author: Fabian Hueske <fh...@apache.org>
AuthorDate: Fri Sep 13 14:17:54 2019 +0200
Rebuild website
---
content/blog/feed.xml | 55 ++++
content/blog/index.html | 36 ++-
content/blog/page2/index.html | 38 +--
content/blog/page3/index.html | 40 +--
content/blog/page4/index.html | 40 +--
content/blog/page5/index.html | 40 +--
content/blog/page6/index.html | 40 +--
content/blog/page7/index.html | 39 ++-
content/blog/page8/index.html | 42 +--
content/blog/page9/index.html | 28 ++
.../feature/2019/09/13/state-processor-api.html | 282 +++++++++++++++++++++
.../application-my-app-state-processor-api.png | Bin 0 -> 49938 bytes
.../database-my-app-state-processor-api.png | Bin 0 -> 50174 bytes
content/index.html | 8 +-
content/zh/index.html | 8 +-
15 files changed, 566 insertions(+), 130 deletions(-)
diff --git a/content/blog/feed.xml b/content/blog/feed.xml
index 2113d6f..1410119 100644
--- a/content/blog/feed.xml
+++ b/content/blog/feed.xml
@@ -7,6 +7,61 @@
<atom:link href="https://flink.apache.org/blog/feed.xml" rel="self" type="application/rss+xml" />
<item>
+<title>The State Processor API: How to Read, write and modify the state of Flink applications</title>
+<description><p>Whether you are running Apache Flink<sup>Ⓡ</sup> in production or evaluated Flink as a computation framework in the past, you’ve probably found yourself asking the question: How can I access, write or update state in a Flink savepoint? Ask no more! <a href="https://flink.apache.org/news/2019/08/22/release-1.9.0.html">Apache Flink 1.9.0</a> introduces the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.9/de [...]
+
+<p>In this post, we explain why this feature is a big step for Flink, what you can use it for, and how to use it. Finally, we will discuss the future of the State Processor API and how it aligns with our plans to evolve Flink into a system for <a href="https://flink.apache.org/news/2019/02/13/unified-batch-streaming-blink.html">unified batch and stream processing</a>.</p>
+
+<h2 id="stateful-stream-processing-with-apache-flink-until-flink-19">Stateful Stream Processing with Apache Flink until Flink 1.9</h2>
+
+<p>All non-trivial stream processing applications are stateful and most of them are designed to run for months or years. Over time, many of them accumulate a lot of valuable state that can be very expensive or even impossible to rebuild if it gets lost due to a failure. In order to guarantee the consistency and durability of application state, Flink featured a sophisticated checkpointing and recovery mechanism from very early on. With every release, the Flink community has added mo [...]
+
+<p>However, a feature that was commonly requested by Flink users was the ability to access the state of an application “from the outside”. This request was motivated by the need to validate or debug the state of an application, to migrate the state of an application to another application, to evolve an application from the Heap State Backend to the RocksDB State Backend, or to import the initial state of an application from an external system like a relational database.</p>
+
+<p>Despite all those convincing reasons to expose application state externally, your access options have been fairly limited until now. Flink’s Queryable State feature only supports key-lookups (point queries) and does not guarantee the consistency of returned values (the value of a key might be different before and after an application recovered from a failure). Moreover, queryable state cannot be used to add or modify the state of an application. Also, savepoints, which are consi [...]
+
+<h2 id="reading-and-writing-application-state-with-the-state-processor-api">Reading and Writing Application State with the State Processor API</h2>
+
+<p>The State Processor API that comes with Flink 1.9 is a true game-changer in how you can work with application state! In a nutshell, it extends the DataSet API with Input and OutputFormats to read and write savepoint or checkpoint data. Due to the <a href="https://ci.apache.org/projects/flink/flink-docs-master/dev/table/common.html#integration-with-datastream-and-dataset-api">interoperability of DataSet and Table API</a>, you can even use relational Table AP [...]
+
+<p>For example, you can take a savepoint of a running stream processing application and analyze it with a DataSet batch program to verify that the application behaves correctly. Or you can read a batch of data from any store, preprocess it, and write the result to a savepoint that you use to bootstrap the state of a streaming application. It’s also possible to fix inconsistent state entries now. Finally, the State Processor API opens up many ways to evolve a stateful application th [...]
+
+<h2 id="mapping-application-state-to-datasets">Mapping Application State to DataSets</h2>
+
+<p>The State Processor API maps the state of a streaming application to one or more data sets that can be separately processed. In order to be able to use the API, you need to understand how this mapping works.</p>
+
+<p>But let’s first have a look at what a stateful Flink job looks like. A Flink job is composed of operators, typically one or more source operators, a few operators for the actual processing, and one or more sink operators. Each operator runs in parallel in one or more tasks and can work with different types of state. An operator can have zero, one, or more <em>“operator states”</em> which are organized as lists that are scoped to the operator’s tasks. If the operator [...]
+
+<p>The following figure shows the application “MyApp” which consists of three operators called “Src”, “Proc”, and “Snk”. Src has one operator state (os1), Proc has one operator state (os2) and two keyed states (ks1, ks2) and Snk is stateless.</p>
+
+<p style="display: block; text-align: center; margin-top: 20px; margin-bottom: 20px">
+ <img src="/img/blog/2019-09-13-state-processor-api-blog/application-my-app-state-processor-api.png" width="600px" alt="Application: My App" />
+</p>
+
+<p>A savepoint or checkpoint of MyApp consists of the data of all states, organized in a way that the states of each task can be restored. When processing the data of a savepoint (or checkpoint) with a batch job, we need a mental model that maps the data of the individual tasks’ states into data sets or tables. In fact, we can think of a savepoint as a database. Every operator (identified by its UID) represents a namespace. Each operator state of an operator is mapped to a dedicate [...]
+
+<p style="display: block; text-align: center; margin-top: 20px; margin-bottom: 20px">
+ <img src="/img/blog/2019-09-13-state-processor-api-blog/database-my-app-state-processor-api.png" width="600px" alt="Database: My App" />
+</p>
+
+<p>The figure shows how the values of Src’s operator state are mapped to a table with one column and five rows, one row for all list entries across all parallel tasks of Src. Operator state os2 of the operator “Proc” is similarly mapped to an individual table. The keyed states ks1 and ks2 are combined to a single table with three columns, one for the key, one for ks1 and one for ks2. The keyed table holds one row for each distinct key of both keyed states. Since the operator “Snk” [...]
+
+<p>The State Processor API now offers methods to create, load, and write a savepoint. You can read a DataSet from a loaded savepoint or convert a DataSet into a state and add it to a savepoint. DataSets can be processed with the full feature set of the DataSet API. With these building blocks, all of the before-mentioned use cases (and more) can be addressed. Please have a look at the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/libs/state_process [...]
+
+<h2 id="why-dataset-api">Why DataSet API?</h2>
+
+<p>In case you are familiar with <a href="https://flink.apache.org/roadmap.html">Flink’s roadmap</a>, you might be surprised that the State Processor API is based on the DataSet API. The Flink community plans to extend the DataStream API with the concept of <em>BoundedStreams</em> and deprecate the DataSet API. When designing this feature, we also evaluated the DataStream API or Table API but neither could provide the right feature set yet. Since w [...]
+
+<h2 id="summary">Summary</h2>
+
+<p>Flink users have requested a feature to access and modify the state of streaming applications from the outside for a long time. With the State Processor API, Flink 1.9.0 finally exposes application state as a data format that can be manipulated. This feature opens up many new possibilities for how users can maintain and manage Flink streaming applications, including arbitrary evolution of stream applications and exporting and bootstrapping of application state. To put it concise [...]
+</description>
+<pubDate>Fri, 13 Sep 2019 14:00:00 +0200</pubDate>
+<link>https://flink.apache.org/feature/2019/09/13/state-processor-api.html</link>
+<guid isPermaLink="true">/feature/2019/09/13/state-processor-api.html</guid>
+</item>
+
+<item>
<title>Apache Flink 1.8.2 Released</title>
<description><p>The Apache Flink community released the second bugfix version of the Apache Flink 1.8 series.</p>
diff --git a/content/blog/index.html b/content/blog/index.html
index da7d5ca..106ff0f 100644
--- a/content/blog/index.html
+++ b/content/blog/index.html
@@ -182,6 +182,19 @@
<!-- Blog posts -->
<article>
+ <h2 class="blog-title"><a href="/feature/2019/09/13/state-processor-api.html">The State Processor API: How to Read, write and modify the state of Flink applications</a></h2>
+
+ <p>13 Sep 2019
+ Seth Wiesman (<a href="https://twitter.com/sjwiesman">@sjwiesman</a>) & Fabian Hueske (<a href="https://twitter.com/fhueske">@fhueske</a>)</p>
+
+ <p>This post explores the State Processor API, introduced with Flink 1.9.0, why this feature is a big step for Flink, what you can use it for, how to use it and explores some future directions that align the feature with Apache Flink's evolution into a system for unified batch and stream processing.</p>
+
+ <p><a href="/feature/2019/09/13/state-processor-api.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
<h2 class="blog-title"><a href="/news/2019/09/11/release-1.8.2.html">Apache Flink 1.8.2 Released</a></h2>
<p>11 Sep 2019
@@ -305,19 +318,6 @@
<hr>
- <article>
- <h2 class="blog-title"><a href="/2019/05/03/pulsar-flink.html">When Flink & Pulsar Come Together</a></h2>
-
- <p>03 May 2019
- Sijie Guo (<a href="https://twitter.com/sijieg">@sijieg</a>)</p>
-
- <p>Apache Flink and Apache Pulsar are distributed data processing systems. When combined, they offer elastic data processing at large scale. This post describes how Pulsar and Flink can work together to provide a seamless developer experience.</p>
-
- <p><a href="/2019/05/03/pulsar-flink.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -350,6 +350,16 @@
<ul id="markdown-toc">
+ <li><a href="/feature/2019/09/13/state-processor-api.html">The State Processor API: How to Read, write and modify the state of Flink applications</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/news/2019/09/11/release-1.8.2.html">Apache Flink 1.8.2 Released</a></li>
diff --git a/content/blog/page2/index.html b/content/blog/page2/index.html
index adc4636..3d48cf0 100644
--- a/content/blog/page2/index.html
+++ b/content/blog/page2/index.html
@@ -182,6 +182,19 @@
<!-- Blog posts -->
<article>
+ <h2 class="blog-title"><a href="/2019/05/03/pulsar-flink.html">When Flink & Pulsar Come Together</a></h2>
+
+ <p>03 May 2019
+ Sijie Guo (<a href="https://twitter.com/sijieg">@sijieg</a>)</p>
+
+ <p>Apache Flink and Apache Pulsar are distributed data processing systems. When combined, they offer elastic data processing at large scale. This post describes how Pulsar and Flink can work together to provide a seamless developer experience.</p>
+
+ <p><a href="/2019/05/03/pulsar-flink.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
<h2 class="blog-title"><a href="/news/2019/04/17/sod.html">Apache Flink's Application to Season of Docs</a></h2>
<p>17 Apr 2019
@@ -312,21 +325,6 @@ for more details.</p>
<hr>
- <article>
- <h2 class="blog-title"><a href="/news/2018/12/22/release-1.6.3.html">Apache Flink 1.6.3 Released</a></h2>
-
- <p>22 Dec 2018
- </p>
-
- <p><p>The Apache Flink community released the third bugfix version of the Apache Flink 1.6 series.</p>
-
-</p>
-
- <p><a href="/news/2018/12/22/release-1.6.3.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -359,6 +357,16 @@ for more details.</p>
<ul id="markdown-toc">
+ <li><a href="/feature/2019/09/13/state-processor-api.html">The State Processor API: How to Read, write and modify the state of Flink applications</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/news/2019/09/11/release-1.8.2.html">Apache Flink 1.8.2 Released</a></li>
diff --git a/content/blog/page3/index.html b/content/blog/page3/index.html
index 15e85dd..c0de99f 100644
--- a/content/blog/page3/index.html
+++ b/content/blog/page3/index.html
@@ -182,6 +182,21 @@
<!-- Blog posts -->
<article>
+ <h2 class="blog-title"><a href="/news/2018/12/22/release-1.6.3.html">Apache Flink 1.6.3 Released</a></h2>
+
+ <p>22 Dec 2018
+ </p>
+
+ <p><p>The Apache Flink community released the third bugfix version of the Apache Flink 1.6 series.</p>
+
+</p>
+
+ <p><a href="/news/2018/12/22/release-1.6.3.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
<h2 class="blog-title"><a href="/news/2018/12/21/release-1.7.1.html">Apache Flink 1.7.1 Released</a></h2>
<p>21 Dec 2018
@@ -318,21 +333,6 @@ Please check the <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa
<hr>
- <article>
- <h2 class="blog-title"><a href="/news/2018/07/12/release-1.5.1.html">Apache Flink 1.5.1 Released</a></h2>
-
- <p>12 Jul 2018
- </p>
-
- <p><p>The Apache Flink community released the first bugfix version of the Apache Flink 1.5 series.</p>
-
-</p>
-
- <p><a href="/news/2018/07/12/release-1.5.1.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -365,6 +365,16 @@ Please check the <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa
<ul id="markdown-toc">
+ <li><a href="/feature/2019/09/13/state-processor-api.html">The State Processor API: How to Read, write and modify the state of Flink applications</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/news/2019/09/11/release-1.8.2.html">Apache Flink 1.8.2 Released</a></li>
diff --git a/content/blog/page4/index.html b/content/blog/page4/index.html
index c128f47..074142a 100644
--- a/content/blog/page4/index.html
+++ b/content/blog/page4/index.html
@@ -182,6 +182,21 @@
<!-- Blog posts -->
<article>
+ <h2 class="blog-title"><a href="/news/2018/07/12/release-1.5.1.html">Apache Flink 1.5.1 Released</a></h2>
+
+ <p>12 Jul 2018
+ </p>
+
+ <p><p>The Apache Flink community released the first bugfix version of the Apache Flink 1.5 series.</p>
+
+</p>
+
+ <p><a href="/news/2018/07/12/release-1.5.1.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
<h2 class="blog-title"><a href="/news/2018/05/25/release-1.5.0.html">Apache Flink 1.5.0 Release Announcement</a></h2>
<p>25 May 2018
@@ -315,21 +330,6 @@ what’s coming in Flink 1.4.0 as well as a preview of what the Flink community
<hr>
- <article>
- <h2 class="blog-title"><a href="/news/2017/08/05/release-1.3.2.html">Apache Flink 1.3.2 Released</a></h2>
-
- <p>05 Aug 2017
- </p>
-
- <p><p>The Apache Flink community released the second bugfix version of the Apache Flink 1.3 series.</p>
-
-</p>
-
- <p><a href="/news/2017/08/05/release-1.3.2.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -362,6 +362,16 @@ what’s coming in Flink 1.4.0 as well as a preview of what the Flink community
<ul id="markdown-toc">
+ <li><a href="/feature/2019/09/13/state-processor-api.html">The State Processor API: How to Read, write and modify the state of Flink applications</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/news/2019/09/11/release-1.8.2.html">Apache Flink 1.8.2 Released</a></li>
diff --git a/content/blog/page5/index.html b/content/blog/page5/index.html
index 304772f..93b0e66 100644
--- a/content/blog/page5/index.html
+++ b/content/blog/page5/index.html
@@ -182,6 +182,21 @@
<!-- Blog posts -->
<article>
+ <h2 class="blog-title"><a href="/news/2017/08/05/release-1.3.2.html">Apache Flink 1.3.2 Released</a></h2>
+
+ <p>05 Aug 2017
+ </p>
+
+ <p><p>The Apache Flink community released the second bugfix version of the Apache Flink 1.3 series.</p>
+
+</p>
+
+ <p><a href="/news/2017/08/05/release-1.3.2.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
<h2 class="blog-title"><a href="/features/2017/07/04/flink-rescalable-state.html">A Deep Dive into Rescalable State in Apache Flink</a></h2>
<p>04 Jul 2017 by Stefan Richter (<a href="https://twitter.com/">@StefanRRichter</a>)
@@ -309,21 +324,6 @@
<hr>
- <article>
- <h2 class="blog-title"><a href="/news/2016/12/21/release-1.1.4.html">Apache Flink 1.1.4 Released</a></h2>
-
- <p>21 Dec 2016
- </p>
-
- <p><p>The Apache Flink community released the next bugfix version of the Apache Flink 1.1 series.</p>
-
-</p>
-
- <p><a href="/news/2016/12/21/release-1.1.4.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -356,6 +356,16 @@
<ul id="markdown-toc">
+ <li><a href="/feature/2019/09/13/state-processor-api.html">The State Processor API: How to Read, write and modify the state of Flink applications</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/news/2019/09/11/release-1.8.2.html">Apache Flink 1.8.2 Released</a></li>
diff --git a/content/blog/page6/index.html b/content/blog/page6/index.html
index 67b0bc3..c5f841e 100644
--- a/content/blog/page6/index.html
+++ b/content/blog/page6/index.html
@@ -182,6 +182,21 @@
<!-- Blog posts -->
<article>
+ <h2 class="blog-title"><a href="/news/2016/12/21/release-1.1.4.html">Apache Flink 1.1.4 Released</a></h2>
+
+ <p>21 Dec 2016
+ </p>
+
+ <p><p>The Apache Flink community released the next bugfix version of the Apache Flink 1.1 series.</p>
+
+</p>
+
+ <p><a href="/news/2016/12/21/release-1.1.4.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
<h2 class="blog-title"><a href="/news/2016/12/19/2016-year-in-review.html">Apache Flink in 2016: Year in Review</a></h2>
<p>19 Dec 2016 by Mike Winters
@@ -313,21 +328,6 @@
<hr>
- <article>
- <h2 class="blog-title"><a href="/news/2016/04/14/flink-forward-announce.html">Flink Forward 2016 Call for Submissions Is Now Open</a></h2>
-
- <p>14 Apr 2016 by Aljoscha Krettek (<a href="https://twitter.com/">@aljoscha</a>)
- </p>
-
- <p><p>We are happy to announce that the call for submissions for Flink Forward 2016 is now open! The conference will take place September 12-14, 2016 in Berlin, Germany, bringing together the open source stream processing community. Most Apache Flink committers will attend the conference, making it the ideal venue to learn more about the project and its roadmap and connect with the community.</p>
-
-</p>
-
- <p><a href="/news/2016/04/14/flink-forward-announce.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -360,6 +360,16 @@
<ul id="markdown-toc">
+ <li><a href="/feature/2019/09/13/state-processor-api.html">The State Processor API: How to Read, write and modify the state of Flink applications</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/news/2019/09/11/release-1.8.2.html">Apache Flink 1.8.2 Released</a></li>
diff --git a/content/blog/page7/index.html b/content/blog/page7/index.html
index ec16d03..0c94df1 100644
--- a/content/blog/page7/index.html
+++ b/content/blog/page7/index.html
@@ -182,6 +182,21 @@
<!-- Blog posts -->
<article>
+ <h2 class="blog-title"><a href="/news/2016/04/14/flink-forward-announce.html">Flink Forward 2016 Call for Submissions Is Now Open</a></h2>
+
+ <p>14 Apr 2016 by Aljoscha Krettek (<a href="https://twitter.com/">@aljoscha</a>)
+ </p>
+
+ <p><p>We are happy to announce that the call for submissions for Flink Forward 2016 is now open! The conference will take place September 12-14, 2016 in Berlin, Germany, bringing together the open source stream processing community. Most Apache Flink committers will attend the conference, making it the ideal venue to learn more about the project and its roadmap and connect with the community.</p>
+
+</p>
+
+ <p><a href="/news/2016/04/14/flink-forward-announce.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
<h2 class="blog-title"><a href="/news/2016/04/06/cep-monitoring.html">Introducing Complex Event Processing (CEP) with Apache Flink</a></h2>
<p>06 Apr 2016 by Till Rohrmann (<a href="https://twitter.com/">@stsffap</a>)
@@ -309,20 +324,6 @@
<hr>
- <article>
- <h2 class="blog-title"><a href="/news/2015/09/16/off-heap-memory.html">Off-heap Memory in Apache Flink and the curious JIT compiler</a></h2>
-
- <p>16 Sep 2015 by Stephan Ewen (<a href="https://twitter.com/">@stephanewen</a>)
- </p>
-
- <p><p>Running data-intensive code in the JVM and making it well-behaved is tricky. Systems that put billions of data objects naively onto the JVM heap face unpredictable OutOfMemoryErrors and Garbage Collection stalls. Of course, you still want to to keep your data in memory as much as possible, for speed and responsiveness of the processing applications. In that context, "off-heap" has become almost something like a magic word to solve these problems.</p>
-<p>In this blog post, we will look at how Flink exploits off-heap memory. The feature is part of the upcoming release, but you can try it out with the latest nightly builds. We will also give a few interesting insights into the behavior for Java's JIT compiler for highly optimized methods and loops.</p></p>
-
- <p><a href="/news/2015/09/16/off-heap-memory.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -355,6 +356,16 @@
<ul id="markdown-toc">
+ <li><a href="/feature/2019/09/13/state-processor-api.html">The State Processor API: How to Read, write and modify the state of Flink applications</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/news/2019/09/11/release-1.8.2.html">Apache Flink 1.8.2 Released</a></li>
diff --git a/content/blog/page8/index.html b/content/blog/page8/index.html
index dac95bb..549f30a 100644
--- a/content/blog/page8/index.html
+++ b/content/blog/page8/index.html
@@ -182,6 +182,20 @@
<!-- Blog posts -->
<article>
+ <h2 class="blog-title"><a href="/news/2015/09/16/off-heap-memory.html">Off-heap Memory in Apache Flink and the curious JIT compiler</a></h2>
+
+ <p>16 Sep 2015 by Stephan Ewen (<a href="https://twitter.com/">@stephanewen</a>)
+ </p>
+
+ <p><p>Running data-intensive code in the JVM and making it well-behaved is tricky. Systems that put billions of data objects naively onto the JVM heap face unpredictable OutOfMemoryErrors and Garbage Collection stalls. Of course, you still want to to keep your data in memory as much as possible, for speed and responsiveness of the processing applications. In that context, "off-heap" has become almost something like a magic word to solve these problems.</p>
+<p>In this blog post, we will look at how Flink exploits off-heap memory. The feature is part of the upcoming release, but you can try it out with the latest nightly builds. We will also give a few interesting insights into the behavior for Java's JIT compiler for highly optimized methods and loops.</p></p>
+
+ <p><a href="/news/2015/09/16/off-heap-memory.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
<h2 class="blog-title"><a href="/news/2015/09/03/flink-forward.html">Announcing Flink Forward 2015</a></h2>
<p>03 Sep 2015
@@ -323,24 +337,6 @@ release is a preview release that contains known issues.</p>
<hr>
- <article>
- <h2 class="blog-title"><a href="/news/2015/03/02/february-2015-in-flink.html">February 2015 in the Flink community</a></h2>
-
- <p>02 Mar 2015
- </p>
-
- <p><p>February might be the shortest month of the year, but this does not
-mean that the Flink community has not been busy adding features to the
-system and fixing bugs. Here’s a rundown of the activity in the Flink
-community last month.</p>
-
-</p>
-
- <p><a href="/news/2015/03/02/february-2015-in-flink.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -373,6 +369,16 @@ community last month.</p>
<ul id="markdown-toc">
+ <li><a href="/feature/2019/09/13/state-processor-api.html">The State Processor API: How to Read, write and modify the state of Flink applications</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/news/2019/09/11/release-1.8.2.html">Apache Flink 1.8.2 Released</a></li>
diff --git a/content/blog/page9/index.html b/content/blog/page9/index.html
index 70b7571..9f95737 100644
--- a/content/blog/page9/index.html
+++ b/content/blog/page9/index.html
@@ -182,6 +182,24 @@
<!-- Blog posts -->
<article>
+ <h2 class="blog-title"><a href="/news/2015/03/02/february-2015-in-flink.html">February 2015 in the Flink community</a></h2>
+
+ <p>02 Mar 2015
+ </p>
+
+ <p><p>February might be the shortest month of the year, but this does not
+mean that the Flink community has not been busy adding features to the
+system and fixing bugs. Here’s a rundown of the activity in the Flink
+community last month.</p>
+
+</p>
+
+ <p><a href="/news/2015/03/02/february-2015-in-flink.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
<h2 class="blog-title"><a href="/news/2015/02/09/streaming-example.html">Introducing Flink Streaming</a></h2>
<p>09 Feb 2015
@@ -355,6 +373,16 @@ academic and open source project that Flink originates from.</p>
<ul id="markdown-toc">
+ <li><a href="/feature/2019/09/13/state-processor-api.html">The State Processor API: How to Read, write and modify the state of Flink applications</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/news/2019/09/11/release-1.8.2.html">Apache Flink 1.8.2 Released</a></li>
diff --git a/content/feature/2019/09/13/state-processor-api.html b/content/feature/2019/09/13/state-processor-api.html
new file mode 100644
index 0000000..a2cbfa4
--- /dev/null
+++ b/content/feature/2019/09/13/state-processor-api.html
@@ -0,0 +1,282 @@
+<!DOCTYPE html>
+<html lang="en">
+ <head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+ <!-- The above 3 meta tags *must* come first in the head; any other head content must come *after* these tags -->
+ <title>Apache Flink: The State Processor API: How to Read, write and modify the state of Flink applications</title>
+ <link rel="shortcut icon" href="/favicon.ico" type="image/x-icon">
+ <link rel="icon" href="/favicon.ico" type="image/x-icon">
+
+ <!-- Bootstrap -->
+ <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.4.1/css/bootstrap.min.css">
+ <link rel="stylesheet" href="/css/flink.css">
+ <link rel="stylesheet" href="/css/syntax.css">
+
+ <!-- Blog RSS feed -->
+ <link href="/blog/feed.xml" rel="alternate" type="application/rss+xml" title="Apache Flink Blog: RSS feed" />
+
+ <!-- jQuery (necessary for Bootstrap's JavaScript plugins) -->
+ <!-- We need to load Jquery in the header for custom google analytics event tracking-->
+ <script src="/js/jquery.min.js"></script>
+
+ <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries -->
+ <!-- WARNING: Respond.js doesn't work if you view the page via file:// -->
+ <!--[if lt IE 9]>
+ <script src="https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js"></script>
+ <script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js"></script>
+ <![endif]-->
+ </head>
+ <body>
+
+
+ <!-- Main content. -->
+ <div class="container">
+ <div class="row">
+
+
+ <div id="sidebar" class="col-sm-3">
+
+
+<!-- Top navbar. -->
+ <nav class="navbar navbar-default">
+ <!-- The logo. -->
+ <div class="navbar-header">
+ <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#bs-example-navbar-collapse-1">
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+ <div class="navbar-logo">
+ <a href="/">
+ <img alt="Apache Flink" src="/img/flink-header-logo.svg" width="147px" height="73px">
+ </a>
+ </div>
+ </div><!-- /.navbar-header -->
+
+ <!-- The navigation links. -->
+ <div class="collapse navbar-collapse" id="bs-example-navbar-collapse-1">
+ <ul class="nav navbar-nav navbar-main">
+
+ <!-- First menu section explains visitors what Flink is -->
+
+ <!-- What is Stream Processing? -->
+ <!--
+ <li><a href="/streamprocessing1.html">What is Stream Processing?</a></li>
+ -->
+
+ <!-- What is Flink? -->
+ <li><a href="/flink-architecture.html">What is Apache Flink?</a></li>
+
+
+
+ <!-- Use cases -->
+ <li><a href="/usecases.html">Use Cases</a></li>
+
+ <!-- Powered by -->
+ <li><a href="/poweredby.html">Powered By</a></li>
+
+ <!-- FAQ -->
+ <li><a href="/faq.html">FAQ</a></li>
+
+
+ <!-- Second menu section aims to support Flink users -->
+
+ <!-- Downloads -->
+ <li><a href="/downloads.html">Downloads</a></li>
+
+ <!-- Quickstart -->
+ <li>
+ <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.9/getting-started/tutorials/local_setup.html" target="_blank">Tutorials <small><span class="glyphicon glyphicon-new-window"></span></small></a>
+ </li>
+
+ <!-- Documentation -->
+ <li class="dropdown">
+ <a class="dropdown-toggle" data-toggle="dropdown" href="#">Documentation<span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li><a href="https://ci.apache.org/projects/flink/flink-docs-release-1.9" target="_blank">1.9 (Latest stable release) <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+ <li><a href="https://ci.apache.org/projects/flink/flink-docs-master" target="_blank">Master (Latest Snapshot) <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+ </ul>
+ </li>
+
+ <!-- getting help -->
+ <li><a href="/gettinghelp.html">Getting Help</a></li>
+
+ <!-- Blog -->
+ <li><a href="/blog/"><b>Flink Blog</b></a></li>
+
+
+
+ <!-- Third menu section aim to support community and contributors -->
+
+ <!-- Community -->
+ <li><a href="/community.html">Community & Project Info</a></li>
+
+ <!-- Roadmap -->
+ <li><a href="/roadmap.html">Roadmap</a></li>
+
+ <!-- Contribute -->
+ <li><a href="/contributing/how-to-contribute.html">How to Contribute</a></li>
+
+
+ <!-- GitHub -->
+ <li>
+ <a href="https://github.com/apache/flink" target="_blank">Flink on GitHub <small><span class="glyphicon glyphicon-new-window"></span></small></a>
+ </li>
+
+
+
+ <!-- Language Switcher -->
+ <li>
+
+
+ <a href="/zh/feature/2019/09/13/state-processor-api.html">中文版</a>
+
+
+ </li>
+
+ </ul>
+
+ <ul class="nav navbar-nav navbar-bottom">
+ <hr />
+
+ <!-- Twitter -->
+ <li><a href="https://twitter.com/apacheflink" target="_blank">@ApacheFlink <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+
+ <!-- Visualizer -->
+ <li class=" hidden-md hidden-sm"><a href="/visualizer/" target="_blank">Plan Visualizer <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+
+ <hr />
+
+ <li><a href="https://apache.org" target="_blank">Apache Software Foundation <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+
+ <li>
+ <style>
+ .smalllinks:link {
+ display: inline-block !important; background: none; padding-top: 0px; padding-bottom: 0px; padding-right: 0px; min-width: 75px;
+ }
+ </style>
+
+ <a class="smalllinks" href="https://www.apache.org/licenses/" target="_blank">License</a> <small><span class="glyphicon glyphicon-new-window"></span></small>
+
+ <a class="smalllinks" href="https://www.apache.org/security/" target="_blank">Security</a> <small><span class="glyphicon glyphicon-new-window"></span></small>
+
+ <a class="smalllinks" href="https://www.apache.org/foundation/sponsorship.html" target="_blank">Donate</a> <small><span class="glyphicon glyphicon-new-window"></span></small>
+
+ <a class="smalllinks" href="https://www.apache.org/foundation/thanks.html" target="_blank">Thanks</a> <small><span class="glyphicon glyphicon-new-window"></span></small>
+ </li>
+
+ </ul>
+ </div><!-- /.navbar-collapse -->
+ </nav>
+
+ </div>
+ <div class="col-sm-9">
+ <div class="row-fluid">
+ <div class="col-sm-12">
+ <div class="row">
+ <h1>The State Processor API: How to Read, write and modify the state of Flink applications</h1>
+
+ <article>
+ <p>13 Sep 2019 Seth Wiesman (<a href="https://twitter.com/sjwiesman">@sjwiesman</a>) & Fabian Hueske (<a href="https://twitter.com/fhueske">@fhueske</a>)</p>
+
+<p>Whether you are running Apache Flink<sup>Ⓡ</sup> in production or evaluated Flink as a computation framework in the past, you’ve probably found yourself asking the question: How can I access, write or update state in a Flink savepoint? Ask no more! <a href="https://flink.apache.org/news/2019/08/22/release-1.9.0.html">Apache Flink 1.9.0</a> introduces the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/libs/state_processor_api.html">State Processor API</a>, a p [...]
+
+<p>In this post, we explain why this feature is a big step for Flink, what you can use it for, and how to use it. Finally, we will discuss the future of the State Processor API and how it aligns with our plans to evolve Flink into a system for <a href="https://flink.apache.org/news/2019/02/13/unified-batch-streaming-blink.html">unified batch and stream processing</a>.</p>
+
+<h2 id="stateful-stream-processing-with-apache-flink-until-flink-19">Stateful Stream Processing with Apache Flink until Flink 1.9</h2>
+
+<p>All non-trivial stream processing applications are stateful and most of them are designed to run for months or years. Over time, many of them accumulate a lot of valuable state that can be very expensive or even impossible to rebuild if it gets lost due to a failure. In order to guarantee the consistency and durability of application state, Flink featured a sophisticated checkpointing and recovery mechanism from very early on. With every release, the Flink community has added more and [...]
+
+<p>However, a feature that was commonly requested by Flink users was the ability to access the state of an application “from the outside”. This request was motivated by the need to validate or debug the state of an application, to migrate the state of an application to another application, to evolve an application from the Heap State Backend to the RocksDB State Backend, or to import the initial state of an application from an external system like a relational database.</p>
+
+<p>Despite all those convincing reasons to expose application state externally, your access options have been fairly limited until now. Flink’s Queryable State feature only supports key-lookups (point queries) and does not guarantee the consistency of returned values (the value of a key might be different before and after an application recovered from a failure). Moreover, queryable state cannot be used to add or modify the state of an application. Also, savepoints, which are consistent [...]
+
+<h2 id="reading-and-writing-application-state-with-the-state-processor-api">Reading and Writing Application State with the State Processor API</h2>
+
+<p>The State Processor API that comes with Flink 1.9 is a true game-changer in how you can work with application state! In a nutshell, it extends the DataSet API with Input and OutputFormats to read and write savepoint or checkpoint data. Due to the <a href="https://ci.apache.org/projects/flink/flink-docs-master/dev/table/common.html#integration-with-datastream-and-dataset-api">interoperability of DataSet and Table API</a>, you can even use relational Table API or SQL queries to analyze [...]
+
+<p>For example, you can take a savepoint of a running stream processing application and analyze it with a DataSet batch program to verify that the application behaves correctly. Or you can read a batch of data from any store, preprocess it, and write the result to a savepoint that you use to bootstrap the state of a streaming application. It’s also possible to fix inconsistent state entries now. Finally, the State Processor API opens up many ways to evolve a stateful application that wer [...]
+
+<h2 id="mapping-application-state-to-datasets">Mapping Application State to DataSets</h2>
+
+<p>The State Processor API maps the state of a streaming application to one or more data sets that can be separately processed. In order to be able to use the API, you need to understand how this mapping works.</p>
+
+<p>But let’s first have a look at what a stateful Flink job looks like. A Flink job is composed of operators, typically one or more source operators, a few operators for the actual processing, and one or more sink operators. Each operator runs in parallel in one or more tasks and can work with different types of state. An operator can have zero, one, or more <em>“operator states”</em> which are organized as lists that are scoped to the operator’s tasks. If the operator is applied on a ke [...]
+
+<p>The following figure shows the application “MyApp” which consists of three operators called “Src”, “Proc”, and “Snk”. Src has one operator state (os1), Proc has one operator state (os2) and two keyed states (ks1, ks2) and Snk is stateless.</p>
+
+<p style="display: block; text-align: center; margin-top: 20px; margin-bottom: 20px">
+ <img src="/img/blog/2019-09-13-state-processor-api-blog/application-my-app-state-processor-api.png" width="600px" alt="Application: My App" />
+</p>
+
+<p>A savepoint or checkpoint of MyApp consists of the data of all states, organized in a way that the states of each task can be restored. When processing the data of a savepoint (or checkpoint) with a batch job, we need a mental model that maps the data of the individual tasks’ states into data sets or tables. In fact, we can think of a savepoint as a database. Every operator (identified by its UID) represents a namespace. Each operator state of an operator is mapped to a dedicated tabl [...]
+
+<p style="display: block; text-align: center; margin-top: 20px; margin-bottom: 20px">
+ <img src="/img/blog/2019-09-13-state-processor-api-blog/database-my-app-state-processor-api.png" width="600px" alt="Database: My App" />
+</p>
+
+<p>The figure shows how the values of Src’s operator state are mapped to a table with one column and five rows, one row for all list entries across all parallel tasks of Src. Operator state os2 of the operator “Proc” is similarly mapped to an individual table. The keyed states ks1 and ks2 are combined to a single table with three columns, one for the key, one for ks1 and one for ks2. The keyed table holds one row for each distinct key of both keyed states. Since the operator “Snk” does n [...]
+
+<p>The State Processor API now offers methods to create, load, and write a savepoint. You can read a DataSet from a loaded savepoint or convert a DataSet into a state and add it to a savepoint. DataSets can be processed with the full feature set of the DataSet API. With these building blocks, all of the before-mentioned use cases (and more) can be addressed. Please have a look at the <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/libs/state_processor_api.html">d [...]
+
+<h2 id="why-dataset-api">Why DataSet API?</h2>
+
+<p>In case you are familiar with <a href="https://flink.apache.org/roadmap.html">Flink’s roadmap</a>, you might be surprised that the State Processor API is based on the DataSet API. The Flink community plans to extend the DataStream API with the concept of <em>BoundedStreams</em> and deprecate the DataSet API. When designing this feature, we also evaluated the DataStream API or Table API but neither could provide the right feature set yet. Since we didn’t want to block this feature on t [...]
+
+<h2 id="summary">Summary</h2>
+
+<p>Flink users have requested a feature to access and modify the state of streaming applications from the outside for a long time. With the State Processor API, Flink 1.9.0 finally exposes application state as a data format that can be manipulated. This feature opens up many new possibilities for how users can maintain and manage Flink streaming applications, including arbitrary evolution of stream applications and exporting and bootstrapping of application state. To put it concisely, th [...]
+
+ </article>
+ </div>
+
+ <div class="row">
+ <div id="disqus_thread"></div>
+ <script type="text/javascript">
+ /* * * CONFIGURATION VARIABLES: EDIT BEFORE PASTING INTO YOUR WEBPAGE * * */
+ var disqus_shortname = 'stratosphere-eu'; // required: replace example with your forum shortname
+
+ /* * * DON'T EDIT BELOW THIS LINE * * */
+ (function() {
+ var dsq = document.createElement('script'); dsq.type = 'text/javascript'; dsq.async = true;
+ dsq.src = '//' + disqus_shortname + '.disqus.com/embed.js';
+ (document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(dsq);
+ })();
+ </script>
+ </div>
+ </div>
+</div>
+ </div>
+ </div>
+
+ <hr />
+
+ <div class="row">
+ <div class="footer text-center col-sm-12">
+ <p>Copyright © 2014-2019 <a href="http://apache.org">The Apache Software Foundation</a>. All Rights Reserved.</p>
+ <p>Apache Flink, Flink®, Apache®, the squirrel logo, and the Apache feather logo are either registered trademarks or trademarks of The Apache Software Foundation.</p>
+ <p><a href="/privacy-policy.html">Privacy Policy</a> · <a href="/blog/feed.xml">RSS feed</a></p>
+ </div>
+ </div>
+ </div><!-- /.container -->
+
+ <!-- Include all compiled plugins (below), or include individual files as needed -->
+ <script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.4/js/bootstrap.min.js"></script>
+ <script src="https://cdnjs.cloudflare.com/ajax/libs/jquery.matchHeight/0.7.0/jquery.matchHeight-min.js"></script>
+ <script src="/js/codetabs.js"></script>
+ <script src="/js/stickysidebar.js"></script>
+
+ <!-- Google Analytics -->
+ <script>
+ (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
+ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
+ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
+ })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
+
+ ga('create', 'UA-52545728-1', 'auto');
+ ga('send', 'pageview');
+ </script>
+ </body>
+</html>
diff --git a/content/img/blog/2019-09-13-state-processor-api-blog/application-my-app-state-processor-api.png b/content/img/blog/2019-09-13-state-processor-api-blog/application-my-app-state-processor-api.png
new file mode 100644
index 0000000..ca6ef6c
Binary files /dev/null and b/content/img/blog/2019-09-13-state-processor-api-blog/application-my-app-state-processor-api.png differ
diff --git a/content/img/blog/2019-09-13-state-processor-api-blog/database-my-app-state-processor-api.png b/content/img/blog/2019-09-13-state-processor-api-blog/database-my-app-state-processor-api.png
new file mode 100644
index 0000000..31642fb
Binary files /dev/null and b/content/img/blog/2019-09-13-state-processor-api-blog/database-my-app-state-processor-api.png differ
diff --git a/content/index.html b/content/index.html
index 3a7c9ac..aff2564 100644
--- a/content/index.html
+++ b/content/index.html
@@ -524,6 +524,9 @@
<dl>
+ <dt> <a href="/feature/2019/09/13/state-processor-api.html">The State Processor API: How to Read, write and modify the state of Flink applications</a></dt>
+ <dd>This post explores the State Processor API, introduced with Flink 1.9.0, why this feature is a big step for Flink, what you can use it for, how to use it and explores some future directions that align the feature with Apache Flink's evolution into a system for unified batch and stream processing.</dd>
+
<dt> <a href="/news/2019/09/11/release-1.8.2.html">Apache Flink 1.8.2 Released</a></dt>
<dd><p>The Apache Flink community released the second bugfix version of the Apache Flink 1.8 series.</p>
@@ -540,11 +543,6 @@
<dt> <a href="/2019/07/23/flink-network-stack-2.html">Flink Network Stack Vol. 2: Monitoring, Metrics, and that Backpressure Thing</a></dt>
<dd>In a previous blog post, we presented how Flink’s network stack works from the high-level abstractions to the low-level details. This second post discusses monitoring network-related metrics to identify backpressure or bottlenecks in throughput and latency.</dd>
-
- <dt> <a href="/news/2019/07/02/release-1.8.1.html">Apache Flink 1.8.1 Released</a></dt>
- <dd><p>The Apache Flink community released the first bugfix version of the Apache Flink 1.8 series.</p>
-
-</dd>
</dl>
diff --git a/content/zh/index.html b/content/zh/index.html
index 5677c97..0465de0 100644
--- a/content/zh/index.html
+++ b/content/zh/index.html
@@ -515,6 +515,9 @@
<dl>
+ <dt> <a href="/feature/2019/09/13/state-processor-api.html">The State Processor API: How to Read, write and modify the state of Flink applications</a></dt>
+ <dd>This post explores the State Processor API, introduced with Flink 1.9.0, why this feature is a big step for Flink, what you can use it for, how to use it and explores some future directions that align the feature with Apache Flink's evolution into a system for unified batch and stream processing.</dd>
+
<dt> <a href="/news/2019/09/11/release-1.8.2.html">Apache Flink 1.8.2 Released</a></dt>
<dd><p>The Apache Flink community released the second bugfix version of the Apache Flink 1.8 series.</p>
@@ -531,11 +534,6 @@
<dt> <a href="/2019/07/23/flink-network-stack-2.html">Flink Network Stack Vol. 2: Monitoring, Metrics, and that Backpressure Thing</a></dt>
<dd>In a previous blog post, we presented how Flink’s network stack works from the high-level abstractions to the low-level details. This second post discusses monitoring network-related metrics to identify backpressure or bottlenecks in throughput and latency.</dd>
-
- <dt> <a href="/news/2019/07/02/release-1.8.1.html">Apache Flink 1.8.1 Released</a></dt>
- <dd><p>The Apache Flink community released the first bugfix version of the Apache Flink 1.8 series.</p>
-
-</dd>
</dl>