You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by fh...@apache.org on 2020/01/29 17:34:40 UTC
[flink-web] branch asf-site updated (0c80e86 -> 3a953f8)
This is an automated email from the ASF dual-hosted git repository.
fhueske pushed a change to branch asf-site
in repository https://gitbox.apache.org/repos/asf/flink-web.git.
from 0c80e86 Add Stateful Functions repository to website
new fb51490 Add "State Unlocked" blog post
new 3a953f8 Rebuild website
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
...ocked-interacting-with-state-in-apache-flink.md | 253 +++++++++++
content/blog/feed.xml | 239 +++++++++++
content/blog/index.html | 36 +-
content/blog/page10/index.html | 25 ++
content/blog/page2/index.html | 36 +-
content/blog/page3/index.html | 38 +-
content/blog/page4/index.html | 38 +-
content/blog/page5/index.html | 38 +-
content/blog/page6/index.html | 40 +-
content/blog/page7/index.html | 40 +-
content/blog/page8/index.html | 40 +-
content/blog/page9/index.html | 40 +-
...state-in-flink-state-processor-api-visual-1.png | Bin 0 -> 52723 bytes
...state-in-flink-state-processor-api-visual-2.png | Bin 0 -> 46207 bytes
.../managing-state-in-flink-visual-1.png | Bin 0 -> 419621 bytes
.../managing-state-in-flink-visual-2.png | Bin 0 -> 408204 bytes
content/index.html | 8 +-
...ked-interacting-with-state-in-apache-flink.html | 470 +++++++++++++++++++++
content/zh/index.html | 8 +-
...state-in-flink-state-processor-api-visual-1.png | Bin 0 -> 52723 bytes
...state-in-flink-state-processor-api-visual-2.png | Bin 0 -> 46207 bytes
.../managing-state-in-flink-visual-1.png | Bin 0 -> 419621 bytes
.../managing-state-in-flink-visual-2.png | Bin 0 -> 408204 bytes
23 files changed, 1210 insertions(+), 139 deletions(-)
create mode 100755 _posts/2020-01-29-state-unlocked-interacting-with-state-in-apache-flink.md
create mode 100755 content/img/blog/2020-01-29-state-unlocked-interacting-with-state-in-apache-flink/managing-state-in-flink-state-processor-api-visual-1.png
create mode 100755 content/img/blog/2020-01-29-state-unlocked-interacting-with-state-in-apache-flink/managing-state-in-flink-state-processor-api-visual-2.png
create mode 100755 content/img/blog/2020-01-29-state-unlocked-interacting-with-state-in-apache-flink/managing-state-in-flink-visual-1.png
create mode 100755 content/img/blog/2020-01-29-state-unlocked-interacting-with-state-in-apache-flink/managing-state-in-flink-visual-2.png
create mode 100644 content/news/2020/01/29/state-unlocked-interacting-with-state-in-apache-flink.html
create mode 100755 img/blog/2020-01-29-state-unlocked-interacting-with-state-in-apache-flink/managing-state-in-flink-state-processor-api-visual-1.png
create mode 100755 img/blog/2020-01-29-state-unlocked-interacting-with-state-in-apache-flink/managing-state-in-flink-state-processor-api-visual-2.png
create mode 100755 img/blog/2020-01-29-state-unlocked-interacting-with-state-in-apache-flink/managing-state-in-flink-visual-1.png
create mode 100755 img/blog/2020-01-29-state-unlocked-interacting-with-state-in-apache-flink/managing-state-in-flink-visual-2.png
[flink-web] 02/02: Rebuild website
Posted by fh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
fhueske pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/flink-web.git
commit 3a953f8faf1ac137c55cd9a5d195f209f43eeab3
Author: Fabian Hueske <fh...@apache.org>
AuthorDate: Wed Jan 29 18:34:00 2020 +0100
Rebuild website
---
content/blog/feed.xml | 239 +++++++++++
content/blog/index.html | 36 +-
content/blog/page10/index.html | 25 ++
content/blog/page2/index.html | 36 +-
content/blog/page3/index.html | 38 +-
content/blog/page4/index.html | 38 +-
content/blog/page5/index.html | 38 +-
content/blog/page6/index.html | 40 +-
content/blog/page7/index.html | 40 +-
content/blog/page8/index.html | 40 +-
content/blog/page9/index.html | 40 +-
...state-in-flink-state-processor-api-visual-1.png | Bin 0 -> 52723 bytes
...state-in-flink-state-processor-api-visual-2.png | Bin 0 -> 46207 bytes
.../managing-state-in-flink-visual-1.png | Bin 0 -> 419621 bytes
.../managing-state-in-flink-visual-2.png | Bin 0 -> 408204 bytes
content/index.html | 8 +-
...ked-interacting-with-state-in-apache-flink.html | 470 +++++++++++++++++++++
content/zh/index.html | 8 +-
18 files changed, 957 insertions(+), 139 deletions(-)
diff --git a/content/blog/feed.xml b/content/blog/feed.xml
index 742368a..9a4424e 100644
--- a/content/blog/feed.xml
+++ b/content/blog/feed.xml
@@ -7,6 +7,245 @@
<atom:link href="https://flink.apache.org/blog/feed.xml" rel="self" type="application/rss+xml" />
<item>
+<title>State Unlocked: Interacting with State in Apache Flink</title>
+<description><h1 id="introduction">Introduction</h1>
+
+<p>With stateful stream-processing becoming the norm for complex event-driven applications and real-time analytics, <a href="https://flink.apache.org/">Apache Flink</a> is often the backbone for running business logic and managing an organization’s most valuable asset — its data — as application state in Flink.</p>
+
+<p>In order to provide a state-of-the-art experience to Flink developers, the Apache Flink community makes significant efforts to provide the safety and future-proof guarantees organizations need while managing state in Flink. In particular, Flink developers should have sufficient means to access and modify their state, as well as making bootstrapping state with existing data from external systems a piece-of-cake. These efforts span multiple Flink major releases and consist of the [...]
+
+<ol>
+ <li>Evolvable state schema in Apache Flink</li>
+ <li>Flexibility in swapping state backends, and</li>
+ <li>The State processor API, an offline tool to read, write and modify state in Flink</li>
+</ol>
+
+<p>This post discusses the community’s efforts related to state management in Flink, provides some practical examples of how the different features and APIs can be utilized and covers some future ideas for new and improved ways of managing state in Apache Flink.</p>
+
+<h1 id="stream-processing-what-is-state">Stream processing: What is State?</h1>
+
+<p>To set the tone for the remaining of the post, let us first try to explain the very definition of state in stream processing. When it comes to stateful stream processing, state comprises of the information that an application or stream processing engine will remember across events and streams as more realtime (unbounded) and/or offline (bounded) data flow through the system. Most trivial applications are inherently stateful; even the example of a simple COUNT operation, whereby [...]
+
+<p>To better understand how Flink manages state, one can think of Flink like a three-layered state abstraction, as illustrated in the diagram below.</p>
+
+<center>
+<img src="/img/blog/2020-01-29-state-unlocked-interacting-with-state-in-apache-flink/managing-state-in-flink-visual-1.png" width="600px" alt="State in Apache Flink" />
+</center>
+<p><br /></p>
+
+<p>On the top layer, sits the Flink user code, for example, a <code>KeyedProcessFunction</code> that contains some value state. This is a simple variable whose value state annotations makes it automatically fault-tolerant, re-scalable and queryable by the runtime. These variables are backed by the configured state backend that sits either on-heap or on-disk (RocksDB State Backend) and provides data locality, proximity to the computation and speed when it comes to per-re [...]
+
+<p>A savepoint is a snapshot of the distributed, global state of an application at a logical point-in-time and is stored in an external distributed file system or blob storage such as HDFS, or S3. Upon upgrading an application or implementing a code change — such as adding a new operator or changing a field — the Flink job can restart by re-loading the application state from the savepoint into the state backend, making it local and available for the computation and continue proces [...]
+
+<center>
+<img src="/img/blog/2020-01-29-state-unlocked-interacting-with-state-in-apache-flink/managing-state-in-flink-visual-2.png" width="600px" alt="State in Apache Flink" />
+</center>
+<p><br /></p>
+
+<div class="alert alert-info">
+ It is important to remember here that <b>state is one of the most valuable components of a Flink application</b> carrying all the information about both where you are now and where you are going. State is among the most long-lived components in a Flink service since it can be carried across jobs, operators, configurations, new features and bug fixes.
+</div>
+
+<h1 id="schema-evolution-with-apache-flink">Schema Evolution with Apache Flink</h1>
+
+<p>In the previous section, we explained how state is stored and persisted in a Flink application. Let’s now take a look at what happens when evolving state in a stateful Flink streaming application becomes necessary.</p>
+
+<p>Imagine an Apache Flink application that implements a <code>KeyedProcessFunction</code> and contains some <code>ValueState</code>. As illustrated below, within the state descriptor, when registering the type, Flink users specify their <code>TypeInformation</code> that informs Flink about how to serialize the bytes and represents Flink’s internal type system, used to serialize data when shipped across the network or stored in state backends. Fl [...]
+
+<h2 id="state-registration-with-built-in-serialization-in-apache-flink">State registration with built-in serialization in Apache Flink</h2>
+
+<div class="highlight"><pre><code class="language-java"><span class="kd">public</span> <span class="kd">class</span> <span class="nc">MyFunction</span> <span class="kd">extends</span> <span class="n">KeyedProcessFunction</span><span class="o">&lt;</span><span class="n">Key</span><span class=& [...]
+<span class="err"></span>
+ <span class="kd">private</span> <span class="kd">transient</span> <span class="n">ValueState</span><span class="o">&lt;</span><span class="n">MyState</span><span class="o">&gt;</span> <span class="n">valueState</span><span class="o">;</span>
+<span class="err"></span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="nf">open</span><span class="o">(</span><span class="n">Configuration</span> <span class="n">parameters</span><span class="o">)</span> <span class="o">{</span>
+ <span class="n">ValueStateDescriptor</span><span class="o">&lt;</span><span class="n">MyState</span><span class="o">&gt;</span> <span class="n">descriptor</span> <span class="o">=</span>
+ <span class="k">new</span> <span class="n">ValueStateDescriptor</span><span class="o">&lt;&gt;(</span><span class="s">&quot;my-state&quot;</span><span class="o">,</span> <span class="n">TypeInformation</span><span class="o">.</span><span class="na">of</span><span class="o">(< [...]
+<span class="err"></span>
+ <span class="n">valueState</span> <span class="o">=</span> <span class="n">getRuntimeContext</span><span class="o">().</span><span class="na">getState</span><span class="o">(</span><span class="n">descriptor</span><span class="o">);</span>
+ <span class="o">}</span>
+<span class="o">}</span></code></pre></div>
+
+<p>Typically, evolving the schema of an application’s state happens because of some business logic change (adding or dropping fields or changing data types). In all cases, the schema is determined by means of its serializer, and can be thought of in terms of an alter table statement when compared with a database. When a state variable is first introduced it is like running a <code>CREATE_TABLE</code> command, there is a lot of freedom with its execution. However, having [...]
+
+<p><a href="https://flink.apache.org/downloads.html#apache-flink-182">Flink 1.8</a> comes with built-in support for <a href="https://avro.apache.org/">Apache Avro</a> (specifically the <a href="https://avro.apache.org/docs/1.7.7/spec.html">1.7.7 specification</a>) and evolves state schema according to Avro specifications by adding and removing types or even by swapping between generic and specific Avro record types.& [...]
+
+<p>In <a href="https://flink.apache.org/downloads.html#apache-flink-191">Flink 1.9</a> the community added support for schema evolution for POJOs, including the ability to remove existing fields from POJO types or add new fields. The POJO schema evolution tends to be less flexible — when compared to Avro — since it is not possible to change neither the declared field types nor the class name of a POJO type, including its namespace.</p>
+
+<p>With the community’s efforts related to schema evolution, Flink developers can now expect out-of-the-box support for both Avro and POJO formats, with backwards compatibility for all Flink state backends. Future work revolves around adding support for Scala Case Classes, Tuples and other formats. Make sure to subscribe to the <a href="https://flink.apache.org/community.html">Flink mailing list</a> to contribute and stay on top of any upcoming additions in th [...]
+
+<h2 id="peeking-under-the-hood">Peeking Under the Hood</h2>
+
+<p>Now that we have explained how schema evolution in Flink works, let’s describe the challenges of performing schema serialization with Flink under the hood. Flink considers state as a core part of its API stability, in a way that developers should always be able to take a savepoint from one version of Flink and restart it on the next. With schema evolution, every migration needs to be backwards compatible and also compatible with the different state backends. While in the Flink c [...]
+
+<p>For instance, the heap state backend supports lazy serialization and eager deserialization, making the per-record code path always working with Java objects, serializing on a background thread. When restoring, Flink will eagerly deserialize all the data and then start the user code. If a developer plugs in a new serializer, the deserialization happens before Flink ever receives the information.</p>
+
+<p>The RocksDB state backend behaves in the exact opposite manner: it supports eager serialization — because of items being stored on disk and RocksDB only consuming byte arrays. RocksDB provides lazy deserialization simply by downloading files to the local disk, making Flink unaware of what the bytes mean until a serializer is registered.</p>
+
+<p>An additional challenge stems from the fact that different versions of user code contain different classes on their classpath making the serializer used to write into a savepoint likely potentially unavailable at runtime.</p>
+
+<p>To overcome the previously mentioned challenges, we introduced what we call <code>TypeSerializerSnapshot</code>. The <code>TypeSerializerSnapshot</code> stores the configuration of the writer serializer in the snapshot. When restoring it will use that configuration to read back the previous state and check its compatibility with the current version. Using such operation allows Flink to:</p>
+
+<ul>
+ <li>Read the configuration used to write out a snapshot</li>
+ <li>Consume the new user code</li>
+ <li>Check if both items above are compatible</li>
+ <li>Consume the bytes from the snapshot and move forward or alert the user otherwise</li>
+</ul>
+
+<div class="highlight"><pre><code class="language-java"><span class="kd">public</span> <span class="kd">interface</span> <span class="nc">TypeSerializerSnapshot</span><span class="o">&lt;</span><span class="n">T</span><span class="o">&gt;</span> <span class="o">{</span>
+<span class="err"></span>
+ <span class="kt">int</span> <span class="nf">getCurrentVersion</span><span class="o">();</span>
+<span class="err"></span>
+ <span class="kt">void</span> <span class="nf">writeSnapshot</span><span class="o">(</span><span class="n">DataOutputView</span> <span class="n">out</span><span class="o">)</span> <span class="kd">throws</span> <span class="n">IOException</span><span class="o">;</span>
+<span class="err"></span>
+ <span class="kt">void</span> <span class="nf">readSnapshot</span><span class="o">(</span>
+ <span class="kt">int</span> <span class="n">readVersion</span><span class="o">,</span>
+ <span class="n">DataInputView</span> <span class="n">in</span><span class="o">,</span>
+ <span class="n">ClassLoader</span> <span class="n">userCodeClassLoader</span><span class="o">)</span> <span class="kd">throws</span> <span class="n">IOException</span><span class="o">;</span>
+<span class="err"></span>
+ <span class="n">TypeSerializer</span><span class="o">&lt;</span><span class="n">T</span><span class="o">&gt;</span> <span class="nf">restoreSerializer</span><span class="o">();</span>
+<span class="err"></span>
+ <span class="n">TypeSerializerSchemaCompatibility</span><span class="o">&lt;</span><span class="n">T</span><span class="o">&gt;</span> <span class="nf">resolveSchemaCompatibility</span><span class="o">(</span>
+ <span class="n">TypeSerializer</span><span class="o">&lt;</span><span class="n">T</span><span class="o">&gt;</span> <span class="n">newSerializer</span><span class="o">);</span>
+<span class="o">}</span></code></pre></div>
+
+<h2 id="implementing-apache-avro-serialization-in-flink">Implementing Apache Avro Serialization in Flink</h2>
+
+<p>Apache Avro is a data serialization format that has very well-defined schema migration semantics and supports both reader and writer schemas. During normal Flink execution the reader and writer schemas will be the same. However, when upgrading an application they may be different and with schema evolution, Flink will be able to migrate objects with their schemas.</p>
+
+<div class="highlight"><pre><code class="language-java"><span class="kd">public</span> <span class="kd">class</span> <span class="nc">AvroSerializerSnapshot</span><span class="o">&lt;</span><span class="n">T</span><span class="o">&gt;</span> <span class="kd">implements</span> <span class= [...]
+ <span class="kd">private</span> <span class="n">Schema</span> <span class="n">runtimeSchema</span><span class="o">;</span>
+ <span class="kd">private</span> <span class="n">Schema</span> <span class="n">previousSchema</span><span class="o">;</span>
+<span class="err"></span>
+ <span class="nd">@SuppressWarnings</span><span class="o">(</span><span class="s">&quot;WeakerAccess&quot;</span><span class="o">)</span>
+ <span class="kd">public</span> <span class="nf">AvroSerializerSnapshot</span><span class="o">()</span> <span class="o">{</span> <span class="o">}</span>
+<span class="err"></span>
+ <span class="n">AvroSerializerSnapshot</span><span class="o">(</span><span class="n">Schema</span> <span class="n">schema</span><span class="o">)</span> <span class="o">{</span>
+ <span class="k">this</span><span class="o">.</span><span class="na">runtimeSchema</span> <span class="o">=</span> <span class="n">schema</span><span class="o">;</span>
+ <span class="o">}</span></code></pre></div>
+
+<p>This is a sketch of our Avro serializer. It uses the provided schemas and delegates to Apache Avro for all (de)-serialization. Let’s take a look at one possible implementation of a <code>TypeSerializerSnapshot</code> that supports schema migration for Avro.</p>
+
+<h1 id="writing-out-the-snapshot">Writing out the snapshot</h1>
+
+<p>When serializing out the snapshot, the snapshot configuration will write two pieces of information; the current snapshot configuration version and the serializer configuration.</p>
+
+<div class="highlight"><pre><code class="language-java"> <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="kt">int</span> <span class="nf">getCurrentVersion</span><span class="o">()</span> <span class="o">{</span>
+ <span class="k">return</span> <span class="mi">1</span><span class="o">;</span>
+ <span class="o">}</span>
+<span class="err"></span>
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="nf">writeSnapshot</span><span class="o">(</span><span class="n">DataOutputView</span> <span class="n">out</span><span class="o">)</span> <span class="kd">throws</span> <span class="n">IOException</span> <span class=& [...]
+ <span class="n">out</span><span class="o">.</span><span class="na">writeUTF</span><span class="o">(</span><span class="n">runtimeSchema</span><span class="o">.</span><span class="na">toString</span><span class="o">(</span><span class="kc">false</span><span class="o">));< [...]
+ <span class="o">}</span></code></pre></div>
+
+<p>The version is used to version the snapshot configuration object itself while the <code>writeSnapshot</code> method writes out all the information we need to understand the current format; the runtime schema.</p>
+
+<div class="highlight"><pre><code class="language-java"> <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="nf">readSnapshot</span><span class="o">(</span>
+ <span class="kt">int</span> <span class="n">readVersion</span><span class="o">,</span>
+ <span class="n">DataInputView</span> <span class="n">in</span><span class="o">,</span>
+ <span class="n">ClassLoader</span> <span class="n">userCodeClassLoader</span><span class="o">)</span> <span class="kd">throws</span> <span class="n">IOException</span> <span class="o">{</span>
+
+ <span class="k">assert</span> <span class="n">readVersion</span> <span class="o">==</span> <span class="mi">1</span><span class="o">;</span>
+ <span class="kd">final</span> <span class="n">String</span> <span class="n">previousSchemaDefinition</span> <span class="o">=</span> <span class="n">in</span><span class="o">.</span><span class="na">readUTF</span><span class="o">();</span>
+ <span class="k">this</span><span class="o">.</span><span class="na">previousSchema</span> <span class="o">=</span> <span class="n">parseAvroSchema</span><span class="o">(</span><span class="n">previousSchemaDefinition</span><span class="o">);</span>
+ <span class="k">this</span><span class="o">.</span><span class="na">runtimeType</span> <span class="o">=</span> <span class="n">findClassOrFallbackToGeneric</span><span class="o">(</span>
+ <span class="n">userCodeClassLoader</span><span class="o">,</span>
+ <span class="n">previousSchema</span><span class="o">.</span><span class="na">getFullName</span><span class="o">());</span>
+<span class="err"></span>
+ <span class="k">this</span><span class="o">.</span><span class="na">runtimeSchema</span> <span class="o">=</span> <span class="n">tryExtractAvroSchema</span><span class="o">(</span><span class="n">userCodeClassLoader</span><span class="o">,</span> <span class="n">runtimeType</span><span [...]
+ <span class="o">}</span></code></pre></div>
+<p>Now when Flink restores it is able to read back in the writer schema used to serialize the data. The current runtime schema is discovered on the class path using some Java reflection magic.</p>
+
+<p>Once we have both of these we can compare them for compatibility. Perhaps nothing has changed and the schemas are compatible as is.</p>
+
+<div class="highlight"><pre><code class="language-java"> <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="n">TypeSerializerSchemaCompatibility</span><span class="o">&lt;</span><span class="n">T</span><span class="o">&gt;</span> <span class="nf">resolveSchemaCompatibility</span><span class="o">(</span>
+ <span class="n">TypeSerializer</span><span class="o">&lt;</span><span class="n">T</span><span class="o">&gt;</span> <span class="n">newSerializer</span><span class="o">)</span> <span class="o">{</span>
+<span class="err"></span>
+ <span class="k">if</span> <span class="o">(!(</span><span class="n">newSerializer</span> <span class="k">instanceof</span> <span class="n">AvroSerializer</span><span class="o">))</span> <span class="o">{</span>
+ <span class="k">return</span> <span class="n">TypeSerializerSchemaCompatibility</span><span class="o">.</span><span class="na">incompatible</span><span class="o">();</span>
+ <span class="o">}</span>
+<span class="err"></span>
+ <span class="k">if</span> <span class="o">(</span><span class="n">Objects</span><span class="o">.</span><span class="na">equals</span><span class="o">(</span><span class="n">previousSchema</span><span class="o">,</span> <span class="n">runtimeSchema</span><span class="o">)) [...]
+ <span class="k">return</span> <span class="n">TypeSerializerSchemaCompatibility</span><span class="o">.</span><span class="na">compatibleAsIs</span><span class="o">();</span>
+ <span class="o">}</span></code></pre></div>
+
+<p>Otherwise, the schemas are compared using Avro’s compatibility checks and they may either be compatible with a migration or incompatible.</p>
+
+<div class="highlight"><pre><code class="language-java"> <span class="kd">final</span> <span class="n">SchemaPairCompatibility</span> <span class="n">compatibility</span> <span class="o">=</span> <span class="n">SchemaCompatibility</span>
+ <span class="o">.</span><span class="na">checkReaderWriterCompatibility</span><span class="o">(</span><span class="n">previousSchema</span><span class="o">,</span> <span class="n">runtimeSchema</span><span class="o">);</span>
+<span class="err"></span>
+ <span class="k">return</span> <span class="nf">avroCompatibilityToFlinkCompatibility</span><span class="o">(</span><span class="n">compatibility</span><span class="o">);</span>
+ <span class="o">}</span></code></pre></div>
+
+<p>If they are compatible with migration then Flink will restore a new serializer that can read the old schema and deserialize into the new runtime type which is in effect a migration.</p>
+
+<div class="highlight"><pre><code class="language-java"> <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="n">TypeSerializer</span><span class="o">&lt;</span><span class="n">T</span><span class="o">&gt;</span> <span class="nf">restoreSerializer</span><span class="o">()</span> <span class="o">{</span>
+ <span class="k">if</span> <span class="o">(</span><span class="n">previousSchema</span> <span class="o">!=</span> <span class="kc">null</span><span class="o">)</span> <span class="o">{</span>
+ <span class="k">return</span> <span class="k">new</span> <span class="n">AvroSerializer</span><span class="o">&lt;&gt;(</span><span class="n">runtimeType</span><span class="o">,</span> <span class="n">runtimeSchema</span><span class="o">,</span> <span class="n">previousSchema</span& [...]
+ <span class="o">}</span> <span class="k">else</span> <span class="o">{</span>
+ <span class="k">return</span> <span class="k">new</span> <span class="n">AvroSerializer</span><span class="o">&lt;&gt;(</span><span class="n">runtimeType</span><span class="o">,</span> <span class="n">runtimeSchema</span><span class="o">,</span> <span class="n">runtimeSchema</span&g [...]
+ <span class="o">}</span>
+ <span class="o">}</span>
+<span class="o">}</span></code></pre></div>
+
+<h1 id="the-state-processor-api-reading-writing-and-modifying-flink-state">The State Processor API: Reading, writing and modifying Flink state</h1>
+
+<p>The State Processor API allows reading from and writing to Flink savepoints. Some of the interesting use cases it can be used for are:</p>
+
+<ul>
+ <li>Analyzing state for interesting patterns</li>
+ <li>Troubleshooting or auditing jobs by checking for state discrepancies</li>
+ <li>Bootstrapping state for new applications</li>
+ <li>Modifying savepoints such as:
+ <ul>
+ <li>Changing the maximum parallelism of a savepoint after deploying a Flink job</li>
+ <li>Introducing breaking schema updates to a Flink application</li>
+ <li>Correcting invalid state in a Flink savepoint</li>
+ </ul>
+ </li>
+</ul>
+
+<p>In a <a href="https://flink.apache.org/feature/2019/09/13/state-processor-api.html">previous blog post</a>, we discussed the State Processor API in detail, the community’s motivation behind introducing the feature in Flink 1.9, what you can use the API for and how you can use it. Essentially, the State Processor API is based around a relational model of mapping your Flink job state to a database, as illustrated in the diagram below. We encourage you to < [...]
+
+<ul>
+ <li>Reading Keyed and Operator State with the State Processor API and</li>
+ <li>Writing and Bootstrapping Keyed and Operator State with the State Processor API</li>
+</ul>
+
+<p>Stay tuned for more details and guidance around this feature of Flink.</p>
+
+<center>
+<img src="/img/blog/2020-01-29-state-unlocked-interacting-with-state-in-apache-flink/managing-state-in-flink-state-processor-api-visual-1.png" width="600px" alt="State Processor API in Apache Flink" />
+</center>
+<p><br /></p>
+
+<center>
+<img src="/img/blog/2020-01-29-state-unlocked-interacting-with-state-in-apache-flink/managing-state-in-flink-state-processor-api-visual-2.png" width="600px" alt="State Processor API in Apache Flink" />
+</center>
+<p><br /></p>
+
+<h1 id="looking-ahead-more-ways-to-interact-with-state-in-flink">Looking ahead: More ways to interact with State in Flink</h1>
+
+<p>There is a lot of discussion happening in the community related to extending the way Flink developers interact with state in their Flink applications. Regarding the State Processor API, some thoughts revolve around further broadening the API’s scope beyond its current ability to read from and write to both keyed and operator state. In upcoming releases, the State processor API will be extended to support both reading from and writing to windows and have a first-class integration [...]
+
+<p>Beyond widening the scope of the State Processor API, the Flink community is discussing a few additional ways to improve the way developers interact with state in Flink. One of them is the proposal for a Unified Savepoint Format (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-41%3A+Unify+Binary+format+for+Keyed+State">FLIP-41</a>) for all keyed state backends. Such improvement aims at introducing a unified binary format across all savepoint [...]
+
+<p>The community is also discussing the ability to have upgradability dry runs in upcoming Flink releases. Having such functionality in Flink allows developers to detect incompatible updates offline without the need of starting a new Flink job from scratch. For example, Flink users will be able to uncover topology or schema incompatibilities upon upgrading a Flink job, without having to load the state back to a running Flink job in the first place. Additionally, with upgradability [...]
+
+<p>With all the exciting new functionality added in Flink 1.9 as well as some solid ideas and discussions around bringing state in Flink to the next level, the community is committed to making state in Apache Flink a fundamental element of the framework, something that is ever-present across versions and upgrades of your application and a component that is a true first-class citizen in Apache Flink. We encourage you to sign up to the <a href="https://flink.apache.org/commu [...]
+</description>
+<pubDate>Wed, 29 Jan 2020 13:00:00 +0100</pubDate>
+<link>https://flink.apache.org/news/2020/01/29/state-unlocked-interacting-with-state-in-apache-flink.html</link>
+<guid isPermaLink="true">/news/2020/01/29/state-unlocked-interacting-with-state-in-apache-flink.html</guid>
+</item>
+
+<item>
<title>Advanced Flink Application Patterns Vol.1: Case Study of a Fraud Detection System</title>
<description><p>In this series of blog posts you will learn about three powerful Flink patterns for building streaming applications:</p>
diff --git a/content/blog/index.html b/content/blog/index.html
index 78ed5ba..0230004 100644
--- a/content/blog/index.html
+++ b/content/blog/index.html
@@ -185,6 +185,19 @@
<!-- Blog posts -->
<article>
+ <h2 class="blog-title"><a href="/news/2020/01/29/state-unlocked-interacting-with-state-in-apache-flink.html">State Unlocked: Interacting with State in Apache Flink</a></h2>
+
+ <p>29 Jan 2020
+ Seth Wiesman (<a href="https://twitter.com/sjwiesman">@sjwiesman</a>)</p>
+
+ <p>This post discusses the efforts of the Flink community as they relate to state management in Apache Flink. We showcase some practical examples of how the different features and APIs can be utilized and cover some future ideas for new and improved ways of managing state in Apache Flink.</p>
+
+ <p><a href="/news/2020/01/29/state-unlocked-interacting-with-state-in-apache-flink.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
<h2 class="blog-title"><a href="/news/2020/01/15/demo-fraud-detection.html">Advanced Flink Application Patterns Vol.1: Case Study of a Fraud Detection System</a></h2>
<p>15 Jan 2020
@@ -310,19 +323,6 @@
<hr>
- <article>
- <h2 class="blog-title"><a href="/2019/07/23/flink-network-stack-2.html">Flink Network Stack Vol. 2: Monitoring, Metrics, and that Backpressure Thing</a></h2>
-
- <p>23 Jul 2019
- Nico Kruber & Piotr Nowojski </p>
-
- <p>In a previous blog post, we presented how Flink’s network stack works from the high-level abstractions to the low-level details. This second post discusses monitoring network-related metrics to identify backpressure or bottlenecks in throughput and latency.</p>
-
- <p><a href="/2019/07/23/flink-network-stack-2.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -355,6 +355,16 @@
<ul id="markdown-toc">
+ <li><a href="/news/2020/01/29/state-unlocked-interacting-with-state-in-apache-flink.html">State Unlocked: Interacting with State in Apache Flink</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/news/2020/01/15/demo-fraud-detection.html">Advanced Flink Application Patterns Vol.1: Case Study of a Fraud Detection System</a></li>
diff --git a/content/blog/page10/index.html b/content/blog/page10/index.html
index e7dd5b0..233304c 100644
--- a/content/blog/page10/index.html
+++ b/content/blog/page10/index.html
@@ -185,6 +185,21 @@
<!-- Blog posts -->
<article>
+ <h2 class="blog-title"><a href="/news/2015/01/06/december-in-flink.html">December 2014 in the Flink community</a></h2>
+
+ <p>06 Jan 2015
+ </p>
+
+ <p><p>This is the first blog post of a “newsletter” like series where we give a summary of the monthly activity in the Flink community. As the Flink project grows, this can serve as a “tl;dr” for people that are not following the Flink dev and user mailing lists, or those that are simply overwhelmed by the traffic.</p>
+
+</p>
+
+ <p><a href="/news/2015/01/06/december-in-flink.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
<h2 class="blog-title"><a href="/news/2014/11/18/hadoop-compatibility.html">Hadoop Compatibility in Flink</a></h2>
<p>18 Nov 2014 by Fabian Hüske (<a href="https://twitter.com/">@fhueske</a>)
@@ -294,6 +309,16 @@ academic and open source project that Flink originates from.</p>
<ul id="markdown-toc">
+ <li><a href="/news/2020/01/29/state-unlocked-interacting-with-state-in-apache-flink.html">State Unlocked: Interacting with State in Apache Flink</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/news/2020/01/15/demo-fraud-detection.html">Advanced Flink Application Patterns Vol.1: Case Study of a Fraud Detection System</a></li>
diff --git a/content/blog/page2/index.html b/content/blog/page2/index.html
index 7e3c83e..23c4723 100644
--- a/content/blog/page2/index.html
+++ b/content/blog/page2/index.html
@@ -185,6 +185,19 @@
<!-- Blog posts -->
<article>
+ <h2 class="blog-title"><a href="/2019/07/23/flink-network-stack-2.html">Flink Network Stack Vol. 2: Monitoring, Metrics, and that Backpressure Thing</a></h2>
+
+ <p>23 Jul 2019
+ Nico Kruber & Piotr Nowojski </p>
+
+ <p>In a previous blog post, we presented how Flink’s network stack works from the high-level abstractions to the low-level details. This second post discusses monitoring network-related metrics to identify backpressure or bottlenecks in throughput and latency.</p>
+
+ <p><a href="/2019/07/23/flink-network-stack-2.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
<h2 class="blog-title"><a href="/news/2019/07/02/release-1.8.1.html">Apache Flink 1.8.1 Released</a></h2>
<p>02 Jul 2019
@@ -311,19 +324,6 @@ for more details.</p>
<hr>
- <article>
- <h2 class="blog-title"><a href="/news/2019/03/06/ffsf-preview.html">What to expect from Flink Forward San Francisco 2019</a></h2>
-
- <p>06 Mar 2019
- Fabian Hueske (<a href="https://twitter.com/fhueske">@fhueske</a>)</p>
-
- <p>The third annual Flink Forward conference in San Francisco is just a few weeks away. Let's see what Flink Forward SF 2019 has in store for the Apache Flink and stream processing communities. This post covers some of its highlights!</p>
-
- <p><a href="/news/2019/03/06/ffsf-preview.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -356,6 +356,16 @@ for more details.</p>
<ul id="markdown-toc">
+ <li><a href="/news/2020/01/29/state-unlocked-interacting-with-state-in-apache-flink.html">State Unlocked: Interacting with State in Apache Flink</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/news/2020/01/15/demo-fraud-detection.html">Advanced Flink Application Patterns Vol.1: Case Study of a Fraud Detection System</a></li>
diff --git a/content/blog/page3/index.html b/content/blog/page3/index.html
index 326c6c9..a8abe5e 100644
--- a/content/blog/page3/index.html
+++ b/content/blog/page3/index.html
@@ -185,6 +185,19 @@
<!-- Blog posts -->
<article>
+ <h2 class="blog-title"><a href="/news/2019/03/06/ffsf-preview.html">What to expect from Flink Forward San Francisco 2019</a></h2>
+
+ <p>06 Mar 2019
+ Fabian Hueske (<a href="https://twitter.com/fhueske">@fhueske</a>)</p>
+
+ <p>The third annual Flink Forward conference in San Francisco is just a few weeks away. Let's see what Flink Forward SF 2019 has in store for the Apache Flink and stream processing communities. This post covers some of its highlights!</p>
+
+ <p><a href="/news/2019/03/06/ffsf-preview.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
<h2 class="blog-title"><a href="/news/2019/02/25/monitoring-best-practices.html">Monitoring Apache Flink Applications 101</a></h2>
<p>25 Feb 2019
@@ -317,21 +330,6 @@ Please check the <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa
<hr>
- <article>
- <h2 class="blog-title"><a href="/news/2018/10/29/release-1.5.5.html">Apache Flink 1.5.5 Released</a></h2>
-
- <p>29 Oct 2018
- </p>
-
- <p><p>The Apache Flink community released the fifth bugfix version of the Apache Flink 1.5 series.</p>
-
-</p>
-
- <p><a href="/news/2018/10/29/release-1.5.5.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -364,6 +362,16 @@ Please check the <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa
<ul id="markdown-toc">
+ <li><a href="/news/2020/01/29/state-unlocked-interacting-with-state-in-apache-flink.html">State Unlocked: Interacting with State in Apache Flink</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/news/2020/01/15/demo-fraud-detection.html">Advanced Flink Application Patterns Vol.1: Case Study of a Fraud Detection System</a></li>
diff --git a/content/blog/page4/index.html b/content/blog/page4/index.html
index f44d3f6..9014a72 100644
--- a/content/blog/page4/index.html
+++ b/content/blog/page4/index.html
@@ -185,6 +185,21 @@
<!-- Blog posts -->
<article>
+ <h2 class="blog-title"><a href="/news/2018/10/29/release-1.5.5.html">Apache Flink 1.5.5 Released</a></h2>
+
+ <p>29 Oct 2018
+ </p>
+
+ <p><p>The Apache Flink community released the fifth bugfix version of the Apache Flink 1.5 series.</p>
+
+</p>
+
+ <p><a href="/news/2018/10/29/release-1.5.5.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
<h2 class="blog-title"><a href="/news/2018/09/20/release-1.6.1.html">Apache Flink 1.6.1 Released</a></h2>
<p>20 Sep 2018
@@ -319,19 +334,6 @@
<hr>
- <article>
- <h2 class="blog-title"><a href="/features/2018/03/01/end-to-end-exactly-once-apache-flink.html">An Overview of End-to-End Exactly-Once Processing in Apache Flink (with Apache Kafka, too!)</a></h2>
-
- <p>01 Mar 2018
- Piotr Nowojski (<a href="https://twitter.com/PiotrNowojski">@PiotrNowojski</a>) & Mike Winters (<a href="https://twitter.com/wints">@wints</a>)</p>
-
- <p>Flink 1.4.0 introduced a new feature that makes it possible to build end-to-end exactly-once applications with Flink and data sources and sinks that support transactions.</p>
-
- <p><a href="/features/2018/03/01/end-to-end-exactly-once-apache-flink.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -364,6 +366,16 @@
<ul id="markdown-toc">
+ <li><a href="/news/2020/01/29/state-unlocked-interacting-with-state-in-apache-flink.html">State Unlocked: Interacting with State in Apache Flink</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/news/2020/01/15/demo-fraud-detection.html">Advanced Flink Application Patterns Vol.1: Case Study of a Fraud Detection System</a></li>
diff --git a/content/blog/page5/index.html b/content/blog/page5/index.html
index 38177fa..e8927b7 100644
--- a/content/blog/page5/index.html
+++ b/content/blog/page5/index.html
@@ -185,6 +185,19 @@
<!-- Blog posts -->
<article>
+ <h2 class="blog-title"><a href="/features/2018/03/01/end-to-end-exactly-once-apache-flink.html">An Overview of End-to-End Exactly-Once Processing in Apache Flink (with Apache Kafka, too!)</a></h2>
+
+ <p>01 Mar 2018
+ Piotr Nowojski (<a href="https://twitter.com/PiotrNowojski">@PiotrNowojski</a>) & Mike Winters (<a href="https://twitter.com/wints">@wints</a>)</p>
+
+ <p>Flink 1.4.0 introduced a new feature that makes it possible to build end-to-end exactly-once applications with Flink and data sources and sinks that support transactions.</p>
+
+ <p><a href="/features/2018/03/01/end-to-end-exactly-once-apache-flink.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
<h2 class="blog-title"><a href="/news/2018/02/15/release-1.4.1.html">Apache Flink 1.4.1 Released</a></h2>
<p>15 Feb 2018
@@ -318,21 +331,6 @@ what’s coming in Flink 1.4.0 as well as a preview of what the Flink community
<hr>
- <article>
- <h2 class="blog-title"><a href="/news/2017/05/16/official-docker-image.html">Introducing Docker Images for Apache Flink</a></h2>
-
- <p>16 May 2017 by Patrick Lucas (Data Artisans) and Ismaël Mejía (Talend) (<a href="https://twitter.com/">@iemejia</a>)
- </p>
-
- <p><p>For some time, the Apache Flink community has provided scripts to build a Docker image to run Flink. Now, starting with version 1.2.1, Flink will have a <a href="https://hub.docker.com/r/_/flink/">Docker image</a> on the Docker Hub. This image is maintained by the Flink community and curated by the <a href="https://github.com/docker-library/official-images">Docker</a> team to ensure it meets the quality standards for container images of the Docker community.</p>
-
-</p>
-
- <p><a href="/news/2017/05/16/official-docker-image.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -365,6 +363,16 @@ what’s coming in Flink 1.4.0 as well as a preview of what the Flink community
<ul id="markdown-toc">
+ <li><a href="/news/2020/01/29/state-unlocked-interacting-with-state-in-apache-flink.html">State Unlocked: Interacting with State in Apache Flink</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/news/2020/01/15/demo-fraud-detection.html">Advanced Flink Application Patterns Vol.1: Case Study of a Fraud Detection System</a></li>
diff --git a/content/blog/page6/index.html b/content/blog/page6/index.html
index 9613a8b..13d76d1 100644
--- a/content/blog/page6/index.html
+++ b/content/blog/page6/index.html
@@ -185,6 +185,21 @@
<!-- Blog posts -->
<article>
+ <h2 class="blog-title"><a href="/news/2017/05/16/official-docker-image.html">Introducing Docker Images for Apache Flink</a></h2>
+
+ <p>16 May 2017 by Patrick Lucas (Data Artisans) and Ismaël Mejía (Talend) (<a href="https://twitter.com/">@iemejia</a>)
+ </p>
+
+ <p><p>For some time, the Apache Flink community has provided scripts to build a Docker image to run Flink. Now, starting with version 1.2.1, Flink will have a <a href="https://hub.docker.com/r/_/flink/">Docker image</a> on the Docker Hub. This image is maintained by the Flink community and curated by the <a href="https://github.com/docker-library/official-images">Docker</a> team to ensure it meets the quality standards for container images of the Docker community.</p>
+
+</p>
+
+ <p><a href="/news/2017/05/16/official-docker-image.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
<h2 class="blog-title"><a href="/news/2017/04/26/release-1.2.1.html">Apache Flink 1.2.1 Released</a></h2>
<p>26 Apr 2017
@@ -312,21 +327,6 @@
<hr>
- <article>
- <h2 class="blog-title"><a href="/news/2016/08/24/ff16-keynotes-panels.html">Flink Forward 2016: Announcing Schedule, Keynotes, and Panel Discussion</a></h2>
-
- <p>24 Aug 2016
- </p>
-
- <p><p>An update for the Flink community: the <a href="http://flink-forward.org/kb_day/day-1/">Flink Forward 2016 schedule</a> is now available online. This year's event will include 2 days of talks from stream processing experts at Google, MapR, Alibaba, Netflix, Cloudera, and more. Following the talks is a full day of hands-on Flink training.</p>
-
-</p>
-
- <p><a href="/news/2016/08/24/ff16-keynotes-panels.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -359,6 +359,16 @@
<ul id="markdown-toc">
+ <li><a href="/news/2020/01/29/state-unlocked-interacting-with-state-in-apache-flink.html">State Unlocked: Interacting with State in Apache Flink</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/news/2020/01/15/demo-fraud-detection.html">Advanced Flink Application Patterns Vol.1: Case Study of a Fraud Detection System</a></li>
diff --git a/content/blog/page7/index.html b/content/blog/page7/index.html
index 921a45a..d15a770 100644
--- a/content/blog/page7/index.html
+++ b/content/blog/page7/index.html
@@ -185,6 +185,21 @@
<!-- Blog posts -->
<article>
+ <h2 class="blog-title"><a href="/news/2016/08/24/ff16-keynotes-panels.html">Flink Forward 2016: Announcing Schedule, Keynotes, and Panel Discussion</a></h2>
+
+ <p>24 Aug 2016
+ </p>
+
+ <p><p>An update for the Flink community: the <a href="http://flink-forward.org/kb_day/day-1/">Flink Forward 2016 schedule</a> is now available online. This year's event will include 2 days of talks from stream processing experts at Google, MapR, Alibaba, Netflix, Cloudera, and more. Following the talks is a full day of hands-on Flink training.</p>
+
+</p>
+
+ <p><a href="/news/2016/08/24/ff16-keynotes-panels.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
<h2 class="blog-title"><a href="/news/2016/08/11/release-1.1.1.html">Flink 1.1.1 Released</a></h2>
<p>11 Aug 2016
@@ -316,21 +331,6 @@
<hr>
- <article>
- <h2 class="blog-title"><a href="/news/2016/02/11/release-0.10.2.html">Flink 0.10.2 Released</a></h2>
-
- <p>11 Feb 2016
- </p>
-
- <p><p>Today, the Flink community released Flink version <strong>0.10.2</strong>, the second bugfix release of the 0.10 series.</p>
-
-</p>
-
- <p><a href="/news/2016/02/11/release-0.10.2.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -363,6 +363,16 @@
<ul id="markdown-toc">
+ <li><a href="/news/2020/01/29/state-unlocked-interacting-with-state-in-apache-flink.html">State Unlocked: Interacting with State in Apache Flink</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/news/2020/01/15/demo-fraud-detection.html">Advanced Flink Application Patterns Vol.1: Case Study of a Fraud Detection System</a></li>
diff --git a/content/blog/page8/index.html b/content/blog/page8/index.html
index 13e04de..423b106 100644
--- a/content/blog/page8/index.html
+++ b/content/blog/page8/index.html
@@ -185,6 +185,21 @@
<!-- Blog posts -->
<article>
+ <h2 class="blog-title"><a href="/news/2016/02/11/release-0.10.2.html">Flink 0.10.2 Released</a></h2>
+
+ <p>11 Feb 2016
+ </p>
+
+ <p><p>Today, the Flink community released Flink version <strong>0.10.2</strong>, the second bugfix release of the 0.10 series.</p>
+
+</p>
+
+ <p><a href="/news/2016/02/11/release-0.10.2.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
<h2 class="blog-title"><a href="/news/2015/12/18/a-year-in-review.html">Flink 2015: A year in review, and a lookout to 2016</a></h2>
<p>18 Dec 2015 by Robert Metzger (<a href="https://twitter.com/">@rmetzger_</a>)
@@ -320,21 +335,6 @@ vertex-centric or gather-sum-apply to Flink dataflows.</p>
<hr>
- <article>
- <h2 class="blog-title"><a href="/news/2015/06/24/announcing-apache-flink-0.9.0-release.html">Announcing Apache Flink 0.9.0</a></h2>
-
- <p>24 Jun 2015
- </p>
-
- <p><p>The Apache Flink community is pleased to announce the availability of the 0.9.0 release. The release is the result of many months of hard work within the Flink community. It contains many new features and improvements which were previewed in the 0.9.0-milestone1 release and have been polished since then. This is the largest Flink release so far.</p>
-
-</p>
-
- <p><a href="/news/2015/06/24/announcing-apache-flink-0.9.0-release.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -367,6 +367,16 @@ vertex-centric or gather-sum-apply to Flink dataflows.</p>
<ul id="markdown-toc">
+ <li><a href="/news/2020/01/29/state-unlocked-interacting-with-state-in-apache-flink.html">State Unlocked: Interacting with State in Apache Flink</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/news/2020/01/15/demo-fraud-detection.html">Advanced Flink Application Patterns Vol.1: Case Study of a Fraud Detection System</a></li>
diff --git a/content/blog/page9/index.html b/content/blog/page9/index.html
index 48a0768..974d65a 100644
--- a/content/blog/page9/index.html
+++ b/content/blog/page9/index.html
@@ -185,6 +185,21 @@
<!-- Blog posts -->
<article>
+ <h2 class="blog-title"><a href="/news/2015/06/24/announcing-apache-flink-0.9.0-release.html">Announcing Apache Flink 0.9.0</a></h2>
+
+ <p>24 Jun 2015
+ </p>
+
+ <p><p>The Apache Flink community is pleased to announce the availability of the 0.9.0 release. The release is the result of many months of hard work within the Flink community. It contains many new features and improvements which were previewed in the 0.9.0-milestone1 release and have been polished since then. This is the largest Flink release so far.</p>
+
+</p>
+
+ <p><a href="/news/2015/06/24/announcing-apache-flink-0.9.0-release.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
<h2 class="blog-title"><a href="/news/2015/05/14/Community-update-April.html">April 2015 in the Flink community</a></h2>
<p>14 May 2015 by Kostas Tzoumas (<a href="https://twitter.com/">@kostas_tzoumas</a>)
@@ -326,21 +341,6 @@ and offers a new API including definition of flexible windows.</p>
<hr>
- <article>
- <h2 class="blog-title"><a href="/news/2015/01/06/december-in-flink.html">December 2014 in the Flink community</a></h2>
-
- <p>06 Jan 2015
- </p>
-
- <p><p>This is the first blog post of a “newsletter” like series where we give a summary of the monthly activity in the Flink community. As the Flink project grows, this can serve as a “tl;dr” for people that are not following the Flink dev and user mailing lists, or those that are simply overwhelmed by the traffic.</p>
-
-</p>
-
- <p><a href="/news/2015/01/06/december-in-flink.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -373,6 +373,16 @@ and offers a new API including definition of flexible windows.</p>
<ul id="markdown-toc">
+ <li><a href="/news/2020/01/29/state-unlocked-interacting-with-state-in-apache-flink.html">State Unlocked: Interacting with State in Apache Flink</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/news/2020/01/15/demo-fraud-detection.html">Advanced Flink Application Patterns Vol.1: Case Study of a Fraud Detection System</a></li>
diff --git a/content/img/blog/2020-01-29-state-unlocked-interacting-with-state-in-apache-flink/managing-state-in-flink-state-processor-api-visual-1.png b/content/img/blog/2020-01-29-state-unlocked-interacting-with-state-in-apache-flink/managing-state-in-flink-state-processor-api-visual-1.png
new file mode 100755
index 0000000..3ec4b5f
Binary files /dev/null and b/content/img/blog/2020-01-29-state-unlocked-interacting-with-state-in-apache-flink/managing-state-in-flink-state-processor-api-visual-1.png differ
diff --git a/content/img/blog/2020-01-29-state-unlocked-interacting-with-state-in-apache-flink/managing-state-in-flink-state-processor-api-visual-2.png b/content/img/blog/2020-01-29-state-unlocked-interacting-with-state-in-apache-flink/managing-state-in-flink-state-processor-api-visual-2.png
new file mode 100755
index 0000000..c5bd780
Binary files /dev/null and b/content/img/blog/2020-01-29-state-unlocked-interacting-with-state-in-apache-flink/managing-state-in-flink-state-processor-api-visual-2.png differ
diff --git a/content/img/blog/2020-01-29-state-unlocked-interacting-with-state-in-apache-flink/managing-state-in-flink-visual-1.png b/content/img/blog/2020-01-29-state-unlocked-interacting-with-state-in-apache-flink/managing-state-in-flink-visual-1.png
new file mode 100755
index 0000000..d78f128
Binary files /dev/null and b/content/img/blog/2020-01-29-state-unlocked-interacting-with-state-in-apache-flink/managing-state-in-flink-visual-1.png differ
diff --git a/content/img/blog/2020-01-29-state-unlocked-interacting-with-state-in-apache-flink/managing-state-in-flink-visual-2.png b/content/img/blog/2020-01-29-state-unlocked-interacting-with-state-in-apache-flink/managing-state-in-flink-visual-2.png
new file mode 100755
index 0000000..6ea5c78
Binary files /dev/null and b/content/img/blog/2020-01-29-state-unlocked-interacting-with-state-in-apache-flink/managing-state-in-flink-visual-2.png differ
diff --git a/content/index.html b/content/index.html
index d80f5fc..3618d34 100644
--- a/content/index.html
+++ b/content/index.html
@@ -557,6 +557,9 @@
<dl>
+ <dt> <a href="/news/2020/01/29/state-unlocked-interacting-with-state-in-apache-flink.html">State Unlocked: Interacting with State in Apache Flink</a></dt>
+ <dd>This post discusses the efforts of the Flink community as they relate to state management in Apache Flink. We showcase some practical examples of how the different features and APIs can be utilized and cover some future ideas for new and improved ways of managing state in Apache Flink.</dd>
+
<dt> <a href="/news/2020/01/15/demo-fraud-detection.html">Advanced Flink Application Patterns Vol.1: Case Study of a Fraud Detection System</a></dt>
<dd>In this series of blog posts you will learn about three powerful Flink patterns for building streaming applications.</dd>
@@ -570,11 +573,6 @@
<dt> <a href="/news/2019/11/25/query-pulsar-streams-using-apache-flink.html">How to query Pulsar Streams using Apache Flink</a></dt>
<dd>This blog post discusses the new developments and integrations between Apache Flink and Apache Pulsar and showcases how you can leverage Pulsar’s built-in schema to query Pulsar streams in real time using Apache Flink.</dd>
-
- <dt> <a href="/news/2019/10/18/release-1.9.1.html">Apache Flink 1.9.1 Released</a></dt>
- <dd><p>The Apache Flink community released the first bugfix version of the Apache Flink 1.9 series.</p>
-
-</dd>
</dl>
diff --git a/content/news/2020/01/29/state-unlocked-interacting-with-state-in-apache-flink.html b/content/news/2020/01/29/state-unlocked-interacting-with-state-in-apache-flink.html
new file mode 100644
index 0000000..3273cab
--- /dev/null
+++ b/content/news/2020/01/29/state-unlocked-interacting-with-state-in-apache-flink.html
@@ -0,0 +1,470 @@
+<!DOCTYPE html>
+<html lang="en">
+ <head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+ <!-- The above 3 meta tags *must* come first in the head; any other head content must come *after* these tags -->
+ <title>Apache Flink: State Unlocked: Interacting with State in Apache Flink</title>
+ <link rel="shortcut icon" href="/favicon.ico" type="image/x-icon">
+ <link rel="icon" href="/favicon.ico" type="image/x-icon">
+
+ <!-- Bootstrap -->
+ <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.4.1/css/bootstrap.min.css">
+ <link rel="stylesheet" href="/css/flink.css">
+ <link rel="stylesheet" href="/css/syntax.css">
+
+ <!-- Blog RSS feed -->
+ <link href="/blog/feed.xml" rel="alternate" type="application/rss+xml" title="Apache Flink Blog: RSS feed" />
+
+ <!-- jQuery (necessary for Bootstrap's JavaScript plugins) -->
+ <!-- We need to load Jquery in the header for custom google analytics event tracking-->
+ <script src="/js/jquery.min.js"></script>
+
+ <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries -->
+ <!-- WARNING: Respond.js doesn't work if you view the page via file:// -->
+ <!--[if lt IE 9]>
+ <script src="https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js"></script>
+ <script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js"></script>
+ <![endif]-->
+ </head>
+ <body>
+
+
+ <!-- Main content. -->
+ <div class="container">
+ <div class="row">
+
+
+ <div id="sidebar" class="col-sm-3">
+
+
+<!-- Top navbar. -->
+ <nav class="navbar navbar-default">
+ <!-- The logo. -->
+ <div class="navbar-header">
+ <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#bs-example-navbar-collapse-1">
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+ <div class="navbar-logo">
+ <a href="/">
+ <img alt="Apache Flink" src="/img/flink-header-logo.svg" width="147px" height="73px">
+ </a>
+ </div>
+ </div><!-- /.navbar-header -->
+
+ <!-- The navigation links. -->
+ <div class="collapse navbar-collapse" id="bs-example-navbar-collapse-1">
+ <ul class="nav navbar-nav navbar-main">
+
+ <!-- First menu section explains visitors what Flink is -->
+
+ <!-- What is Stream Processing? -->
+ <!--
+ <li><a href="/streamprocessing1.html">What is Stream Processing?</a></li>
+ -->
+
+ <!-- What is Flink? -->
+ <li><a href="/flink-architecture.html">What is Apache Flink?</a></li>
+
+
+
+ <!-- Use cases -->
+ <li><a href="/usecases.html">Use Cases</a></li>
+
+ <!-- Powered by -->
+ <li><a href="/poweredby.html">Powered By</a></li>
+
+ <!-- FAQ -->
+ <li><a href="/faq.html">FAQ</a></li>
+
+
+ <!-- Second menu section aims to support Flink users -->
+
+ <!-- Downloads -->
+ <li><a href="/downloads.html">Downloads</a></li>
+
+ <!-- Getting Started -->
+ <li>
+ <a href="https://ci.apache.org/projects/flink/flink-docs-release-1.9/getting-started/index.html" target="_blank">Getting Started <small><span class="glyphicon glyphicon-new-window"></span></small></a>
+ </li>
+
+ <!-- Documentation -->
+ <li class="dropdown">
+ <a class="dropdown-toggle" data-toggle="dropdown" href="#">Documentation<span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li><a href="https://ci.apache.org/projects/flink/flink-docs-release-1.9" target="_blank">1.9 (Latest stable release) <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+ <li><a href="https://ci.apache.org/projects/flink/flink-docs-master" target="_blank">Master (Latest Snapshot) <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+ </ul>
+ </li>
+
+ <!-- getting help -->
+ <li><a href="/gettinghelp.html">Getting Help</a></li>
+
+ <!-- Blog -->
+ <li class="active"><a href="/blog/"><b>Flink Blog</b></a></li>
+
+ <!-- Ecosystem -->
+ <li><a href="/ecosystem.html">Ecosystem</a></li>
+
+
+
+ <!-- Third menu section aim to support community and contributors -->
+
+ <!-- Community -->
+ <li><a href="/community.html">Community & Project Info</a></li>
+
+ <!-- Roadmap -->
+ <li><a href="/roadmap.html">Roadmap</a></li>
+
+ <!-- Contribute -->
+ <li><a href="/contributing/how-to-contribute.html">How to Contribute</a></li>
+
+
+ <!-- GitHub -->
+ <li>
+ <a href="https://github.com/apache/flink" target="_blank">Flink on GitHub <small><span class="glyphicon glyphicon-new-window"></span></small></a>
+ </li>
+
+
+
+ <!-- Language Switcher -->
+ <li>
+
+
+ <!-- link to the Chinese home page when current is blog page -->
+ <a href="/zh">中文版</a>
+
+
+ </li>
+
+ </ul>
+
+ <ul class="nav navbar-nav navbar-bottom">
+ <hr />
+
+ <!-- Twitter -->
+ <li><a href="https://twitter.com/apacheflink" target="_blank">@ApacheFlink <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+
+ <!-- Visualizer -->
+ <li class=" hidden-md hidden-sm"><a href="/visualizer/" target="_blank">Plan Visualizer <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+
+ <hr />
+
+ <li><a href="https://apache.org" target="_blank">Apache Software Foundation <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+
+ <li>
+ <style>
+ .smalllinks:link {
+ display: inline-block !important; background: none; padding-top: 0px; padding-bottom: 0px; padding-right: 0px; min-width: 75px;
+ }
+ </style>
+
+ <a class="smalllinks" href="https://www.apache.org/licenses/" target="_blank">License</a> <small><span class="glyphicon glyphicon-new-window"></span></small>
+
+ <a class="smalllinks" href="https://www.apache.org/security/" target="_blank">Security</a> <small><span class="glyphicon glyphicon-new-window"></span></small>
+
+ <a class="smalllinks" href="https://www.apache.org/foundation/sponsorship.html" target="_blank">Donate</a> <small><span class="glyphicon glyphicon-new-window"></span></small>
+
+ <a class="smalllinks" href="https://www.apache.org/foundation/thanks.html" target="_blank">Thanks</a> <small><span class="glyphicon glyphicon-new-window"></span></small>
+ </li>
+
+ </ul>
+ </div><!-- /.navbar-collapse -->
+ </nav>
+
+ </div>
+ <div class="col-sm-9">
+ <div class="row-fluid">
+ <div class="col-sm-12">
+ <div class="row">
+ <h1>State Unlocked: Interacting with State in Apache Flink</h1>
+
+ <article>
+ <p>29 Jan 2020 Seth Wiesman (<a href="https://twitter.com/sjwiesman">@sjwiesman</a>)</p>
+
+<h1 id="introduction">Introduction</h1>
+
+<p>With stateful stream-processing becoming the norm for complex event-driven applications and real-time analytics, <a href="https://flink.apache.org/">Apache Flink</a> is often the backbone for running business logic and managing an organization’s most valuable asset — its data — as application state in Flink.</p>
+
+<p>In order to provide a state-of-the-art experience to Flink developers, the Apache Flink community makes significant efforts to provide the safety and future-proof guarantees organizations need while managing state in Flink. In particular, Flink developers should have sufficient means to access and modify their state, as well as making bootstrapping state with existing data from external systems a piece-of-cake. These efforts span multiple Flink major releases and consist of the follow [...]
+
+<ol>
+ <li>Evolvable state schema in Apache Flink</li>
+ <li>Flexibility in swapping state backends, and</li>
+ <li>The State processor API, an offline tool to read, write and modify state in Flink</li>
+</ol>
+
+<p>This post discusses the community’s efforts related to state management in Flink, provides some practical examples of how the different features and APIs can be utilized and covers some future ideas for new and improved ways of managing state in Apache Flink.</p>
+
+<h1 id="stream-processing-what-is-state">Stream processing: What is State?</h1>
+
+<p>To set the tone for the remaining of the post, let us first try to explain the very definition of state in stream processing. When it comes to stateful stream processing, state comprises of the information that an application or stream processing engine will remember across events and streams as more realtime (unbounded) and/or offline (bounded) data flow through the system. Most trivial applications are inherently stateful; even the example of a simple COUNT operation, whereby when c [...]
+
+<p>To better understand how Flink manages state, one can think of Flink like a three-layered state abstraction, as illustrated in the diagram below.</p>
+
+<center>
+<img src="/img/blog/2020-01-29-state-unlocked-interacting-with-state-in-apache-flink/managing-state-in-flink-visual-1.png" width="600px" alt="State in Apache Flink" />
+</center>
+<p><br /></p>
+
+<p>On the top layer, sits the Flink user code, for example, a <code>KeyedProcessFunction</code> that contains some value state. This is a simple variable whose value state annotations makes it automatically fault-tolerant, re-scalable and queryable by the runtime. These variables are backed by the configured state backend that sits either on-heap or on-disk (RocksDB State Backend) and provides data locality, proximity to the computation and speed when it comes to per-record computations. [...]
+
+<p>A savepoint is a snapshot of the distributed, global state of an application at a logical point-in-time and is stored in an external distributed file system or blob storage such as HDFS, or S3. Upon upgrading an application or implementing a code change — such as adding a new operator or changing a field — the Flink job can restart by re-loading the application state from the savepoint into the state backend, making it local and available for the computation and continue processing a [...]
+
+<center>
+<img src="/img/blog/2020-01-29-state-unlocked-interacting-with-state-in-apache-flink/managing-state-in-flink-visual-2.png" width="600px" alt="State in Apache Flink" />
+</center>
+<p><br /></p>
+
+<div class="alert alert-info">
+ It is important to remember here that <b>state is one of the most valuable components of a Flink application</b> carrying all the information about both where you are now and where you are going. State is among the most long-lived components in a Flink service since it can be carried across jobs, operators, configurations, new features and bug fixes.
+</div>
+
+<h1 id="schema-evolution-with-apache-flink">Schema Evolution with Apache Flink</h1>
+
+<p>In the previous section, we explained how state is stored and persisted in a Flink application. Let’s now take a look at what happens when evolving state in a stateful Flink streaming application becomes necessary.</p>
+
+<p>Imagine an Apache Flink application that implements a <code>KeyedProcessFunction</code> and contains some <code>ValueState</code>. As illustrated below, within the state descriptor, when registering the type, Flink users specify their <code>TypeInformation</code> that informs Flink about how to serialize the bytes and represents Flink’s internal type system, used to serialize data when shipped across the network or stored in state backends. Flink’s type system has built-in support for [...]
+
+<h2 id="state-registration-with-built-in-serialization-in-apache-flink">State registration with built-in serialization in Apache Flink</h2>
+
+<div class="highlight"><pre><code class="language-java"><span class="kd">public</span> <span class="kd">class</span> <span class="nc">MyFunction</span> <span class="kd">extends</span> <span class="n">KeyedProcessFunction</span><span class="o"><</span><span class="n">Key</span><span class="o">,</span> <span class="n">Input</span><span class="o">,</span> <span class="n">Output</span><span class="o">></span> <span class="o">{</span>
+<span class="err"></span>
+ <span class="kd">private</span> <span class="kd">transient</span> <span class="n">ValueState</span><span class="o"><</span><span class="n">MyState</span><span class="o">></span> <span class="n">valueState</span><span class="o">;</span>
+<span class="err"></span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="nf">open</span><span class="o">(</span><span class="n">Configuration</span> <span class="n">parameters</span><span class="o">)</span> <span class="o">{</span>
+ <span class="n">ValueStateDescriptor</span><span class="o"><</span><span class="n">MyState</span><span class="o">></span> <span class="n">descriptor</span> <span class="o">=</span>
+ <span class="k">new</span> <span class="n">ValueStateDescriptor</span><span class="o"><>(</span><span class="s">"my-state"</span><span class="o">,</span> <span class="n">TypeInformation</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">MyState</span><span class="o">.</span><span class="na">class</span><span class="o">));</span>
+<span class="err"></span>
+ <span class="n">valueState</span> <span class="o">=</span> <span class="n">getRuntimeContext</span><span class="o">().</span><span class="na">getState</span><span class="o">(</span><span class="n">descriptor</span><span class="o">);</span>
+ <span class="o">}</span>
+<span class="o">}</span></code></pre></div>
+
+<p>Typically, evolving the schema of an application’s state happens because of some business logic change (adding or dropping fields or changing data types). In all cases, the schema is determined by means of its serializer, and can be thought of in terms of an alter table statement when compared with a database. When a state variable is first introduced it is like running a <code>CREATE_TABLE</code> command, there is a lot of freedom with its execution. However, having data in that tabl [...]
+
+<p><a href="https://flink.apache.org/downloads.html#apache-flink-182">Flink 1.8</a> comes with built-in support for <a href="https://avro.apache.org/">Apache Avro</a> (specifically the <a href="https://avro.apache.org/docs/1.7.7/spec.html">1.7.7 specification</a>) and evolves state schema according to Avro specifications by adding and removing types or even by swapping between generic and specific Avro record types.</p>
+
+<p>In <a href="https://flink.apache.org/downloads.html#apache-flink-191">Flink 1.9</a> the community added support for schema evolution for POJOs, including the ability to remove existing fields from POJO types or add new fields. The POJO schema evolution tends to be less flexible — when compared to Avro — since it is not possible to change neither the declared field types nor the class name of a POJO type, including its namespace.</p>
+
+<p>With the community’s efforts related to schema evolution, Flink developers can now expect out-of-the-box support for both Avro and POJO formats, with backwards compatibility for all Flink state backends. Future work revolves around adding support for Scala Case Classes, Tuples and other formats. Make sure to subscribe to the <a href="https://flink.apache.org/community.html">Flink mailing list</a> to contribute and stay on top of any upcoming additions in this space.</p>
+
+<h2 id="peeking-under-the-hood">Peeking Under the Hood</h2>
+
+<p>Now that we have explained how schema evolution in Flink works, let’s describe the challenges of performing schema serialization with Flink under the hood. Flink considers state as a core part of its API stability, in a way that developers should always be able to take a savepoint from one version of Flink and restart it on the next. With schema evolution, every migration needs to be backwards compatible and also compatible with the different state backends. While in the Flink code th [...]
+
+<p>For instance, the heap state backend supports lazy serialization and eager deserialization, making the per-record code path always working with Java objects, serializing on a background thread. When restoring, Flink will eagerly deserialize all the data and then start the user code. If a developer plugs in a new serializer, the deserialization happens before Flink ever receives the information.</p>
+
+<p>The RocksDB state backend behaves in the exact opposite manner: it supports eager serialization — because of items being stored on disk and RocksDB only consuming byte arrays. RocksDB provides lazy deserialization simply by downloading files to the local disk, making Flink unaware of what the bytes mean until a serializer is registered.</p>
+
+<p>An additional challenge stems from the fact that different versions of user code contain different classes on their classpath making the serializer used to write into a savepoint likely potentially unavailable at runtime.</p>
+
+<p>To overcome the previously mentioned challenges, we introduced what we call <code>TypeSerializerSnapshot</code>. The <code>TypeSerializerSnapshot</code> stores the configuration of the writer serializer in the snapshot. When restoring it will use that configuration to read back the previous state and check its compatibility with the current version. Using such operation allows Flink to:</p>
+
+<ul>
+ <li>Read the configuration used to write out a snapshot</li>
+ <li>Consume the new user code</li>
+ <li>Check if both items above are compatible</li>
+ <li>Consume the bytes from the snapshot and move forward or alert the user otherwise</li>
+</ul>
+
+<div class="highlight"><pre><code class="language-java"><span class="kd">public</span> <span class="kd">interface</span> <span class="nc">TypeSerializerSnapshot</span><span class="o"><</span><span class="n">T</span><span class="o">></span> <span class="o">{</span>
+<span class="err"></span>
+ <span class="kt">int</span> <span class="nf">getCurrentVersion</span><span class="o">();</span>
+<span class="err"></span>
+ <span class="kt">void</span> <span class="nf">writeSnapshot</span><span class="o">(</span><span class="n">DataOutputView</span> <span class="n">out</span><span class="o">)</span> <span class="kd">throws</span> <span class="n">IOException</span><span class="o">;</span>
+<span class="err"></span>
+ <span class="kt">void</span> <span class="nf">readSnapshot</span><span class="o">(</span>
+ <span class="kt">int</span> <span class="n">readVersion</span><span class="o">,</span>
+ <span class="n">DataInputView</span> <span class="n">in</span><span class="o">,</span>
+ <span class="n">ClassLoader</span> <span class="n">userCodeClassLoader</span><span class="o">)</span> <span class="kd">throws</span> <span class="n">IOException</span><span class="o">;</span>
+<span class="err"></span>
+ <span class="n">TypeSerializer</span><span class="o"><</span><span class="n">T</span><span class="o">></span> <span class="nf">restoreSerializer</span><span class="o">();</span>
+<span class="err"></span>
+ <span class="n">TypeSerializerSchemaCompatibility</span><span class="o"><</span><span class="n">T</span><span class="o">></span> <span class="nf">resolveSchemaCompatibility</span><span class="o">(</span>
+ <span class="n">TypeSerializer</span><span class="o"><</span><span class="n">T</span><span class="o">></span> <span class="n">newSerializer</span><span class="o">);</span>
+<span class="o">}</span></code></pre></div>
+
+<h2 id="implementing-apache-avro-serialization-in-flink">Implementing Apache Avro Serialization in Flink</h2>
+
+<p>Apache Avro is a data serialization format that has very well-defined schema migration semantics and supports both reader and writer schemas. During normal Flink execution the reader and writer schemas will be the same. However, when upgrading an application they may be different and with schema evolution, Flink will be able to migrate objects with their schemas.</p>
+
+<div class="highlight"><pre><code class="language-java"><span class="kd">public</span> <span class="kd">class</span> <span class="nc">AvroSerializerSnapshot</span><span class="o"><</span><span class="n">T</span><span class="o">></span> <span class="kd">implements</span> <span class="n">TypeSerializerSnapshot</span><span class="o"><</span><span class="n">T</span><span class="o">></span> <span class="o">{</span>
+ <span class="kd">private</span> <span class="n">Schema</span> <span class="n">runtimeSchema</span><span class="o">;</span>
+ <span class="kd">private</span> <span class="n">Schema</span> <span class="n">previousSchema</span><span class="o">;</span>
+<span class="err"></span>
+ <span class="nd">@SuppressWarnings</span><span class="o">(</span><span class="s">"WeakerAccess"</span><span class="o">)</span>
+ <span class="kd">public</span> <span class="nf">AvroSerializerSnapshot</span><span class="o">()</span> <span class="o">{</span> <span class="o">}</span>
+<span class="err"></span>
+ <span class="n">AvroSerializerSnapshot</span><span class="o">(</span><span class="n">Schema</span> <span class="n">schema</span><span class="o">)</span> <span class="o">{</span>
+ <span class="k">this</span><span class="o">.</span><span class="na">runtimeSchema</span> <span class="o">=</span> <span class="n">schema</span><span class="o">;</span>
+ <span class="o">}</span></code></pre></div>
+
+<p>This is a sketch of our Avro serializer. It uses the provided schemas and delegates to Apache Avro for all (de)-serialization. Let’s take a look at one possible implementation of a <code>TypeSerializerSnapshot</code> that supports schema migration for Avro.</p>
+
+<h1 id="writing-out-the-snapshot">Writing out the snapshot</h1>
+
+<p>When serializing out the snapshot, the snapshot configuration will write two pieces of information; the current snapshot configuration version and the serializer configuration.</p>
+
+<div class="highlight"><pre><code class="language-java"> <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="kt">int</span> <span class="nf">getCurrentVersion</span><span class="o">()</span> <span class="o">{</span>
+ <span class="k">return</span> <span class="mi">1</span><span class="o">;</span>
+ <span class="o">}</span>
+<span class="err"></span>
+ <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="nf">writeSnapshot</span><span class="o">(</span><span class="n">DataOutputView</span> <span class="n">out</span><span class="o">)</span> <span class="kd">throws</span> <span class="n">IOException</span> <span class="o">{</span>
+ <span class="n">out</span><span class="o">.</span><span class="na">writeUTF</span><span class="o">(</span><span class="n">runtimeSchema</span><span class="o">.</span><span class="na">toString</span><span class="o">(</span><span class="kc">false</span><span class="o">));</span>
+ <span class="o">}</span></code></pre></div>
+
+<p>The version is used to version the snapshot configuration object itself while the <code>writeSnapshot</code> method writes out all the information we need to understand the current format; the runtime schema.</p>
+
+<div class="highlight"><pre><code class="language-java"> <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="kt">void</span> <span class="nf">readSnapshot</span><span class="o">(</span>
+ <span class="kt">int</span> <span class="n">readVersion</span><span class="o">,</span>
+ <span class="n">DataInputView</span> <span class="n">in</span><span class="o">,</span>
+ <span class="n">ClassLoader</span> <span class="n">userCodeClassLoader</span><span class="o">)</span> <span class="kd">throws</span> <span class="n">IOException</span> <span class="o">{</span>
+
+ <span class="k">assert</span> <span class="n">readVersion</span> <span class="o">==</span> <span class="mi">1</span><span class="o">;</span>
+ <span class="kd">final</span> <span class="n">String</span> <span class="n">previousSchemaDefinition</span> <span class="o">=</span> <span class="n">in</span><span class="o">.</span><span class="na">readUTF</span><span class="o">();</span>
+ <span class="k">this</span><span class="o">.</span><span class="na">previousSchema</span> <span class="o">=</span> <span class="n">parseAvroSchema</span><span class="o">(</span><span class="n">previousSchemaDefinition</span><span class="o">);</span>
+ <span class="k">this</span><span class="o">.</span><span class="na">runtimeType</span> <span class="o">=</span> <span class="n">findClassOrFallbackToGeneric</span><span class="o">(</span>
+ <span class="n">userCodeClassLoader</span><span class="o">,</span>
+ <span class="n">previousSchema</span><span class="o">.</span><span class="na">getFullName</span><span class="o">());</span>
+<span class="err"></span>
+ <span class="k">this</span><span class="o">.</span><span class="na">runtimeSchema</span> <span class="o">=</span> <span class="n">tryExtractAvroSchema</span><span class="o">(</span><span class="n">userCodeClassLoader</span><span class="o">,</span> <span class="n">runtimeType</span><span class="o">);</span>
+ <span class="o">}</span></code></pre></div>
+<p>Now when Flink restores it is able to read back in the writer schema used to serialize the data. The current runtime schema is discovered on the class path using some Java reflection magic.</p>
+
+<p>Once we have both of these we can compare them for compatibility. Perhaps nothing has changed and the schemas are compatible as is.</p>
+
+<div class="highlight"><pre><code class="language-java"> <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="n">TypeSerializerSchemaCompatibility</span><span class="o"><</span><span class="n">T</span><span class="o">></span> <span class="nf">resolveSchemaCompatibility</span><span class="o">(</span>
+ <span class="n">TypeSerializer</span><span class="o"><</span><span class="n">T</span><span class="o">></span> <span class="n">newSerializer</span><span class="o">)</span> <span class="o">{</span>
+<span class="err"></span>
+ <span class="k">if</span> <span class="o">(!(</span><span class="n">newSerializer</span> <span class="k">instanceof</span> <span class="n">AvroSerializer</span><span class="o">))</span> <span class="o">{</span>
+ <span class="k">return</span> <span class="n">TypeSerializerSchemaCompatibility</span><span class="o">.</span><span class="na">incompatible</span><span class="o">();</span>
+ <span class="o">}</span>
+<span class="err"></span>
+ <span class="k">if</span> <span class="o">(</span><span class="n">Objects</span><span class="o">.</span><span class="na">equals</span><span class="o">(</span><span class="n">previousSchema</span><span class="o">,</span> <span class="n">runtimeSchema</span><span class="o">))</span> <span class="o">{</span>
+ <span class="k">return</span> <span class="n">TypeSerializerSchemaCompatibility</span><span class="o">.</span><span class="na">compatibleAsIs</span><span class="o">();</span>
+ <span class="o">}</span></code></pre></div>
+
+<p>Otherwise, the schemas are compared using Avro’s compatibility checks and they may either be compatible with a migration or incompatible.</p>
+
+<div class="highlight"><pre><code class="language-java"> <span class="kd">final</span> <span class="n">SchemaPairCompatibility</span> <span class="n">compatibility</span> <span class="o">=</span> <span class="n">SchemaCompatibility</span>
+ <span class="o">.</span><span class="na">checkReaderWriterCompatibility</span><span class="o">(</span><span class="n">previousSchema</span><span class="o">,</span> <span class="n">runtimeSchema</span><span class="o">);</span>
+<span class="err"></span>
+ <span class="k">return</span> <span class="nf">avroCompatibilityToFlinkCompatibility</span><span class="o">(</span><span class="n">compatibility</span><span class="o">);</span>
+ <span class="o">}</span></code></pre></div>
+
+<p>If they are compatible with migration then Flink will restore a new serializer that can read the old schema and deserialize into the new runtime type which is in effect a migration.</p>
+
+<div class="highlight"><pre><code class="language-java"> <span class="nd">@Override</span>
+ <span class="kd">public</span> <span class="n">TypeSerializer</span><span class="o"><</span><span class="n">T</span><span class="o">></span> <span class="nf">restoreSerializer</span><span class="o">()</span> <span class="o">{</span>
+ <span class="k">if</span> <span class="o">(</span><span class="n">previousSchema</span> <span class="o">!=</span> <span class="kc">null</span><span class="o">)</span> <span class="o">{</span>
+ <span class="k">return</span> <span class="k">new</span> <span class="n">AvroSerializer</span><span class="o"><>(</span><span class="n">runtimeType</span><span class="o">,</span> <span class="n">runtimeSchema</span><span class="o">,</span> <span class="n">previousSchema</span><span class="o">);</span>
+ <span class="o">}</span> <span class="k">else</span> <span class="o">{</span>
+ <span class="k">return</span> <span class="k">new</span> <span class="n">AvroSerializer</span><span class="o"><>(</span><span class="n">runtimeType</span><span class="o">,</span> <span class="n">runtimeSchema</span><span class="o">,</span> <span class="n">runtimeSchema</span><span class="o">);</span>
+ <span class="o">}</span>
+ <span class="o">}</span>
+<span class="o">}</span></code></pre></div>
+
+<h1 id="the-state-processor-api-reading-writing-and-modifying-flink-state">The State Processor API: Reading, writing and modifying Flink state</h1>
+
+<p>The State Processor API allows reading from and writing to Flink savepoints. Some of the interesting use cases it can be used for are:</p>
+
+<ul>
+ <li>Analyzing state for interesting patterns</li>
+ <li>Troubleshooting or auditing jobs by checking for state discrepancies</li>
+ <li>Bootstrapping state for new applications</li>
+ <li>Modifying savepoints such as:
+ <ul>
+ <li>Changing the maximum parallelism of a savepoint after deploying a Flink job</li>
+ <li>Introducing breaking schema updates to a Flink application</li>
+ <li>Correcting invalid state in a Flink savepoint</li>
+ </ul>
+ </li>
+</ul>
+
+<p>In a <a href="https://flink.apache.org/feature/2019/09/13/state-processor-api.html">previous blog post</a>, we discussed the State Processor API in detail, the community’s motivation behind introducing the feature in Flink 1.9, what you can use the API for and how you can use it. Essentially, the State Processor API is based around a relational model of mapping your Flink job state to a database, as illustrated in the diagram below. We encourage you to <a href="https://flink.apache.or [...]
+
+<ul>
+ <li>Reading Keyed and Operator State with the State Processor API and</li>
+ <li>Writing and Bootstrapping Keyed and Operator State with the State Processor API</li>
+</ul>
+
+<p>Stay tuned for more details and guidance around this feature of Flink.</p>
+
+<center>
+<img src="/img/blog/2020-01-29-state-unlocked-interacting-with-state-in-apache-flink/managing-state-in-flink-state-processor-api-visual-1.png" width="600px" alt="State Processor API in Apache Flink" />
+</center>
+<p><br /></p>
+
+<center>
+<img src="/img/blog/2020-01-29-state-unlocked-interacting-with-state-in-apache-flink/managing-state-in-flink-state-processor-api-visual-2.png" width="600px" alt="State Processor API in Apache Flink" />
+</center>
+<p><br /></p>
+
+<h1 id="looking-ahead-more-ways-to-interact-with-state-in-flink">Looking ahead: More ways to interact with State in Flink</h1>
+
+<p>There is a lot of discussion happening in the community related to extending the way Flink developers interact with state in their Flink applications. Regarding the State Processor API, some thoughts revolve around further broadening the API’s scope beyond its current ability to read from and write to both keyed and operator state. In upcoming releases, the State processor API will be extended to support both reading from and writing to windows and have a first-class integration with [...]
+
+<p>Beyond widening the scope of the State Processor API, the Flink community is discussing a few additional ways to improve the way developers interact with state in Flink. One of them is the proposal for a Unified Savepoint Format (<a href="https://cwiki.apache.org/confluence/display/FLINK/FLIP-41%3A+Unify+Binary+format+for+Keyed+State">FLIP-41</a>) for all keyed state backends. Such improvement aims at introducing a unified binary format across all savepoints in all keyed state backend [...]
+
+<p>The community is also discussing the ability to have upgradability dry runs in upcoming Flink releases. Having such functionality in Flink allows developers to detect incompatible updates offline without the need of starting a new Flink job from scratch. For example, Flink users will be able to uncover topology or schema incompatibilities upon upgrading a Flink job, without having to load the state back to a running Flink job in the first place. Additionally, with upgradability dry ru [...]
+
+<p>With all the exciting new functionality added in Flink 1.9 as well as some solid ideas and discussions around bringing state in Flink to the next level, the community is committed to making state in Apache Flink a fundamental element of the framework, something that is ever-present across versions and upgrades of your application and a component that is a true first-class citizen in Apache Flink. We encourage you to sign up to the <a href="https://flink.apache.org/community.html">mai [...]
+
+ </article>
+ </div>
+
+ <div class="row">
+ <div id="disqus_thread"></div>
+ <script type="text/javascript">
+ /* * * CONFIGURATION VARIABLES: EDIT BEFORE PASTING INTO YOUR WEBPAGE * * */
+ var disqus_shortname = 'stratosphere-eu'; // required: replace example with your forum shortname
+
+ /* * * DON'T EDIT BELOW THIS LINE * * */
+ (function() {
+ var dsq = document.createElement('script'); dsq.type = 'text/javascript'; dsq.async = true;
+ dsq.src = '//' + disqus_shortname + '.disqus.com/embed.js';
+ (document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(dsq);
+ })();
+ </script>
+ </div>
+ </div>
+</div>
+ </div>
+ </div>
+
+ <hr />
+
+ <div class="row">
+ <div class="footer text-center col-sm-12">
+ <p>Copyright © 2014-2019 <a href="http://apache.org">The Apache Software Foundation</a>. All Rights Reserved.</p>
+ <p>Apache Flink, Flink®, Apache®, the squirrel logo, and the Apache feather logo are either registered trademarks or trademarks of The Apache Software Foundation.</p>
+ <p><a href="/privacy-policy.html">Privacy Policy</a> · <a href="/blog/feed.xml">RSS feed</a></p>
+ </div>
+ </div>
+ </div><!-- /.container -->
+
+ <!-- Include all compiled plugins (below), or include individual files as needed -->
+ <script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.4/js/bootstrap.min.js"></script>
+ <script src="https://cdnjs.cloudflare.com/ajax/libs/jquery.matchHeight/0.7.0/jquery.matchHeight-min.js"></script>
+ <script src="/js/codetabs.js"></script>
+ <script src="/js/stickysidebar.js"></script>
+
+ <!-- Google Analytics -->
+ <script>
+ (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
+ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
+ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
+ })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
+
+ ga('create', 'UA-52545728-1', 'auto');
+ ga('send', 'pageview');
+ </script>
+ </body>
+</html>
diff --git a/content/zh/index.html b/content/zh/index.html
index e6ec214..2a7d25f 100644
--- a/content/zh/index.html
+++ b/content/zh/index.html
@@ -548,6 +548,9 @@
<dl>
+ <dt> <a href="/news/2020/01/29/state-unlocked-interacting-with-state-in-apache-flink.html">State Unlocked: Interacting with State in Apache Flink</a></dt>
+ <dd>This post discusses the efforts of the Flink community as they relate to state management in Apache Flink. We showcase some practical examples of how the different features and APIs can be utilized and cover some future ideas for new and improved ways of managing state in Apache Flink.</dd>
+
<dt> <a href="/news/2020/01/15/demo-fraud-detection.html">Advanced Flink Application Patterns Vol.1: Case Study of a Fraud Detection System</a></dt>
<dd>In this series of blog posts you will learn about three powerful Flink patterns for building streaming applications.</dd>
@@ -561,11 +564,6 @@
<dt> <a href="/news/2019/11/25/query-pulsar-streams-using-apache-flink.html">How to query Pulsar Streams using Apache Flink</a></dt>
<dd>This blog post discusses the new developments and integrations between Apache Flink and Apache Pulsar and showcases how you can leverage Pulsar’s built-in schema to query Pulsar streams in real time using Apache Flink.</dd>
-
- <dt> <a href="/news/2019/10/18/release-1.9.1.html">Apache Flink 1.9.1 Released</a></dt>
- <dd><p>The Apache Flink community released the first bugfix version of the Apache Flink 1.9 series.</p>
-
-</dd>
</dl>
[flink-web] 01/02: Add "State Unlocked" blog post
Posted by fh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
fhueske pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/flink-web.git
commit fb51490a60d8084b61416607c6d9a8d9c2f91169
Author: Seth Wiesman <sj...@gmail.com>
AuthorDate: Tue Dec 10 12:30:47 2019 -0600
Add "State Unlocked" blog post
This closes #288.
---
...ocked-interacting-with-state-in-apache-flink.md | 253 +++++++++++++++++++++
...state-in-flink-state-processor-api-visual-1.png | Bin 0 -> 52723 bytes
...state-in-flink-state-processor-api-visual-2.png | Bin 0 -> 46207 bytes
.../managing-state-in-flink-visual-1.png | Bin 0 -> 419621 bytes
.../managing-state-in-flink-visual-2.png | Bin 0 -> 408204 bytes
5 files changed, 253 insertions(+)
diff --git a/_posts/2020-01-29-state-unlocked-interacting-with-state-in-apache-flink.md b/_posts/2020-01-29-state-unlocked-interacting-with-state-in-apache-flink.md
new file mode 100755
index 0000000..ee6bb3c
--- /dev/null
+++ b/_posts/2020-01-29-state-unlocked-interacting-with-state-in-apache-flink.md
@@ -0,0 +1,253 @@
+---
+layout: post
+title: "State Unlocked: Interacting with State in Apache Flink"
+date: 2020-01-29 12:00:00
+categories: news
+authors:
+- seth:
+ name: "Seth Wiesman"
+ twitter: "sjwiesman"
+
+
+excerpt: This post discusses the efforts of the Flink community as they relate to state management in Apache Flink. We showcase some practical examples of how the different features and APIs can be utilized and cover some future ideas for new and improved ways of managing state in Apache Flink.
+
+---
+
+# Introduction
+
+With stateful stream-processing becoming the norm for complex event-driven applications and real-time analytics, [Apache Flink](https://flink.apache.org/) is often the backbone for running business logic and managing an organization’s most valuable asset — its data — as application state in Flink.
+
+In order to provide a state-of-the-art experience to Flink developers, the Apache Flink community makes significant efforts to provide the safety and future-proof guarantees organizations need while managing state in Flink. In particular, Flink developers should have sufficient means to access and modify their state, as well as making bootstrapping state with existing data from external systems a piece-of-cake. These efforts span multiple Flink major releases and consist of the following:
+
+1. Evolvable state schema in Apache Flink
+2. Flexibility in swapping state backends, and
+3. The State processor API, an offline tool to read, write and modify state in Flink
+
+This post discusses the community’s efforts related to state management in Flink, provides some practical examples of how the different features and APIs can be utilized and covers some future ideas for new and improved ways of managing state in Apache Flink.
+
+
+# Stream processing: What is State?
+
+To set the tone for the remaining of the post, let us first try to explain the very definition of state in stream processing. When it comes to stateful stream processing, state comprises of the information that an application or stream processing engine will remember across events and streams as more realtime (unbounded) and/or offline (bounded) data flow through the system. Most trivial applications are inherently stateful; even the example of a simple COUNT operation, whereby when coun [...]
+
+To better understand how Flink manages state, one can think of Flink like a three-layered state abstraction, as illustrated in the diagram below.
+
+<center>
+<img src="{{ site.baseurl }}/img/blog/2020-01-29-state-unlocked-interacting-with-state-in-apache-flink/managing-state-in-flink-visual-1.png" width="600px" alt="State in Apache Flink"/>
+</center>
+<br>
+
+On the top layer, sits the Flink user code, for example, a `KeyedProcessFunction` that contains some value state. This is a simple variable whose value state annotations makes it automatically fault-tolerant, re-scalable and queryable by the runtime. These variables are backed by the configured state backend that sits either on-heap or on-disk (RocksDB State Backend) and provides data locality, proximity to the computation and speed when it comes to per-record computations. Finally, when [...]
+
+A savepoint is a snapshot of the distributed, global state of an application at a logical point-in-time and is stored in an external distributed file system or blob storage such as HDFS, or S3. Upon upgrading an application or implementing a code change — such as adding a new operator or changing a field — the Flink job can restart by re-loading the application state from the savepoint into the state backend, making it local and available for the computation and continue processing as i [...]
+
+<center>
+<img src="{{ site.baseurl }}/img/blog/2020-01-29-state-unlocked-interacting-with-state-in-apache-flink/managing-state-in-flink-visual-2.png" width="600px" alt="State in Apache Flink"/>
+</center>
+<br>
+
+<div class="alert alert-info">
+ It is important to remember here that <b>state is one of the most valuable components of a Flink application</b> carrying all the information about both where you are now and where you are going. State is among the most long-lived components in a Flink service since it can be carried across jobs, operators, configurations, new features and bug fixes.
+</div>
+
+# Schema Evolution with Apache Flink
+
+In the previous section, we explained how state is stored and persisted in a Flink application. Let’s now take a look at what happens when evolving state in a stateful Flink streaming application becomes necessary.
+
+Imagine an Apache Flink application that implements a `KeyedProcessFunction` and contains some `ValueState`. As illustrated below, within the state descriptor, when registering the type, Flink users specify their `TypeInformation` that informs Flink about how to serialize the bytes and represents Flink’s internal type system, used to serialize data when shipped across the network or stored in state backends. Flink’s type system has built-in support for all the basic types such as longs, [...]
+
+## State registration with built-in serialization in Apache Flink
+
+```java
+public class MyFunction extends KeyedProcessFunction<Key, Input, Output> {
+
+ private transient ValueState<MyState> valueState;
+
+ public void open(Configuration parameters) {
+ ValueStateDescriptor<MyState> descriptor =
+ new ValueStateDescriptor<>("my-state", TypeInformation.of(MyState.class));
+
+ valueState = getRuntimeContext().getState(descriptor);
+ }
+}
+```
+
+Typically, evolving the schema of an application’s state happens because of some business logic change (adding or dropping fields or changing data types). In all cases, the schema is determined by means of its serializer, and can be thought of in terms of an alter table statement when compared with a database. When a state variable is first introduced it is like running a `CREATE_TABLE` command, there is a lot of freedom with its execution. However, having data in that table (registered [...]
+
+[Flink 1.8](https://flink.apache.org/downloads.html#apache-flink-182) comes with built-in support for [Apache Avro](https://avro.apache.org/) (specifically the [1.7.7 specification](https://avro.apache.org/docs/1.7.7/spec.html)) and evolves state schema according to Avro specifications by adding and removing types or even by swapping between generic and specific Avro record types.
+
+In [Flink 1.9](https://flink.apache.org/downloads.html#apache-flink-191) the community added support for schema evolution for POJOs, including the ability to remove existing fields from POJO types or add new fields. The POJO schema evolution tends to be less flexible — when compared to Avro — since it is not possible to change neither the declared field types nor the class name of a POJO type, including its namespace.
+
+With the community’s efforts related to schema evolution, Flink developers can now expect out-of-the-box support for both Avro and POJO formats, with backwards compatibility for all Flink state backends. Future work revolves around adding support for Scala Case Classes, Tuples and other formats. Make sure to subscribe to the [Flink mailing list](https://flink.apache.org/community.html) to contribute and stay on top of any upcoming additions in this space.
+
+## Peeking Under the Hood
+
+Now that we have explained how schema evolution in Flink works, let’s describe the challenges of performing schema serialization with Flink under the hood. Flink considers state as a core part of its API stability, in a way that developers should always be able to take a savepoint from one version of Flink and restart it on the next. With schema evolution, every migration needs to be backwards compatible and also compatible with the different state backends. While in the Flink code the s [...]
+
+For instance, the heap state backend supports lazy serialization and eager deserialization, making the per-record code path always working with Java objects, serializing on a background thread. When restoring, Flink will eagerly deserialize all the data and then start the user code. If a developer plugs in a new serializer, the deserialization happens before Flink ever receives the information.
+
+The RocksDB state backend behaves in the exact opposite manner: it supports eager serialization — because of items being stored on disk and RocksDB only consuming byte arrays. RocksDB provides lazy deserialization simply by downloading files to the local disk, making Flink unaware of what the bytes mean until a serializer is registered.
+
+An additional challenge stems from the fact that different versions of user code contain different classes on their classpath making the serializer used to write into a savepoint likely potentially unavailable at runtime.
+
+To overcome the previously mentioned challenges, we introduced what we call `TypeSerializerSnapshot`. The `TypeSerializerSnapshot` stores the configuration of the writer serializer in the snapshot. When restoring it will use that configuration to read back the previous state and check its compatibility with the current version. Using such operation allows Flink to:
+
+* Read the configuration used to write out a snapshot
+* Consume the new user code
+* Check if both items above are compatible
+* Consume the bytes from the snapshot and move forward or alert the user otherwise
+
+```java
+public interface TypeSerializerSnapshot<T> {
+
+ int getCurrentVersion();
+
+ void writeSnapshot(DataOutputView out) throws IOException;
+
+ void readSnapshot(
+ int readVersion,
+ DataInputView in,
+ ClassLoader userCodeClassLoader) throws IOException;
+
+ TypeSerializer<T> restoreSerializer();
+
+ TypeSerializerSchemaCompatibility<T> resolveSchemaCompatibility(
+ TypeSerializer<T> newSerializer);
+}
+```
+
+## Implementing Apache Avro Serialization in Flink
+
+Apache Avro is a data serialization format that has very well-defined schema migration semantics and supports both reader and writer schemas. During normal Flink execution the reader and writer schemas will be the same. However, when upgrading an application they may be different and with schema evolution, Flink will be able to migrate objects with their schemas.
+
+```java
+public class AvroSerializerSnapshot<T> implements TypeSerializerSnapshot<T> {
+ private Schema runtimeSchema;
+ private Schema previousSchema;
+
+ @SuppressWarnings("WeakerAccess")
+ public AvroSerializerSnapshot() { }
+
+ AvroSerializerSnapshot(Schema schema) {
+ this.runtimeSchema = schema;
+ }
+```
+
+This is a sketch of our Avro serializer. It uses the provided schemas and delegates to Apache Avro for all (de)-serialization. Let’s take a look at one possible implementation of a `TypeSerializerSnapshot` that supports schema migration for Avro.
+
+
+# Writing out the snapshot
+
+When serializing out the snapshot, the snapshot configuration will write two pieces of information; the current snapshot configuration version and the serializer configuration.
+
+```java
+ @Override
+ public int getCurrentVersion() {
+ return 1;
+ }
+
+ @Override
+ public void writeSnapshot(DataOutputView out) throws IOException {
+ out.writeUTF(runtimeSchema.toString(false));
+ }
+```
+
+The version is used to version the snapshot configuration object itself while the `writeSnapshot` method writes out all the information we need to understand the current format; the runtime schema.
+
+```java
+ @Override
+ public void readSnapshot(
+ int readVersion,
+ DataInputView in,
+ ClassLoader userCodeClassLoader) throws IOException {
+
+ assert readVersion == 1;
+ final String previousSchemaDefinition = in.readUTF();
+ this.previousSchema = parseAvroSchema(previousSchemaDefinition);
+ this.runtimeType = findClassOrFallbackToGeneric(
+ userCodeClassLoader,
+ previousSchema.getFullName());
+
+ this.runtimeSchema = tryExtractAvroSchema(userCodeClassLoader, runtimeType);
+ }
+```
+Now when Flink restores it is able to read back in the writer schema used to serialize the data. The current runtime schema is discovered on the class path using some Java reflection magic.
+
+Once we have both of these we can compare them for compatibility. Perhaps nothing has changed and the schemas are compatible as is.
+
+```java
+ @Override
+ public TypeSerializerSchemaCompatibility<T> resolveSchemaCompatibility(
+ TypeSerializer<T> newSerializer) {
+
+ if (!(newSerializer instanceof AvroSerializer)) {
+ return TypeSerializerSchemaCompatibility.incompatible();
+ }
+
+ if (Objects.equals(previousSchema, runtimeSchema)) {
+ return TypeSerializerSchemaCompatibility.compatibleAsIs();
+ }
+```
+
+Otherwise, the schemas are compared using Avro’s compatibility checks and they may either be compatible with a migration or incompatible.
+
+```java
+ final SchemaPairCompatibility compatibility = SchemaCompatibility
+ .checkReaderWriterCompatibility(previousSchema, runtimeSchema);
+
+ return avroCompatibilityToFlinkCompatibility(compatibility);
+ }
+```
+
+If they are compatible with migration then Flink will restore a new serializer that can read the old schema and deserialize into the new runtime type which is in effect a migration.
+
+```java
+ @Override
+ public TypeSerializer<T> restoreSerializer() {
+ if (previousSchema != null) {
+ return new AvroSerializer<>(runtimeType, runtimeSchema, previousSchema);
+ } else {
+ return new AvroSerializer<>(runtimeType, runtimeSchema, runtimeSchema);
+ }
+ }
+}
+```
+
+# The State Processor API: Reading, writing and modifying Flink state
+
+The State Processor API allows reading from and writing to Flink savepoints. Some of the interesting use cases it can be used for are:
+
+* Analyzing state for interesting patterns
+* Troubleshooting or auditing jobs by checking for state discrepancies
+* Bootstrapping state for new applications
+* Modifying savepoints such as:
+ * Changing the maximum parallelism of a savepoint after deploying a Flink job
+ * Introducing breaking schema updates to a Flink application
+ * Correcting invalid state in a Flink savepoint
+
+In a [previous blog post](https://flink.apache.org/feature/2019/09/13/state-processor-api.html), we discussed the State Processor API in detail, the community’s motivation behind introducing the feature in Flink 1.9, what you can use the API for and how you can use it. Essentially, the State Processor API is based around a relational model of mapping your Flink job state to a database, as illustrated in the diagram below. We encourage you to [read the previous story](https://flink.apache [...]
+
+* Reading Keyed and Operator State with the State Processor API and
+* Writing and Bootstrapping Keyed and Operator State with the State Processor API
+
+Stay tuned for more details and guidance around this feature of Flink.
+
+<center>
+<img src="{{ site.baseurl }}/img/blog/2020-01-29-state-unlocked-interacting-with-state-in-apache-flink/managing-state-in-flink-state-processor-api-visual-1.png" width="600px" alt="State Processor API in Apache Flink"/>
+</center>
+<br>
+
+<center>
+<img src="{{ site.baseurl }}/img/blog/2020-01-29-state-unlocked-interacting-with-state-in-apache-flink/managing-state-in-flink-state-processor-api-visual-2.png" width="600px" alt="State Processor API in Apache Flink"/>
+</center>
+<br>
+
+# Looking ahead: More ways to interact with State in Flink
+
+There is a lot of discussion happening in the community related to extending the way Flink developers interact with state in their Flink applications. Regarding the State Processor API, some thoughts revolve around further broadening the API’s scope beyond its current ability to read from and write to both keyed and operator state. In upcoming releases, the State processor API will be extended to support both reading from and writing to windows and have a first-class integration with Fli [...]
+
+Beyond widening the scope of the State Processor API, the Flink community is discussing a few additional ways to improve the way developers interact with state in Flink. One of them is the proposal for a Unified Savepoint Format ([FLIP-41](https://cwiki.apache.org/confluence/display/FLINK/FLIP-41%3A+Unify+Binary+format+for+Keyed+State)) for all keyed state backends. Such improvement aims at introducing a unified binary format across all savepoints in all keyed state backends, something t [...]
+
+The community is also discussing the ability to have upgradability dry runs in upcoming Flink releases. Having such functionality in Flink allows developers to detect incompatible updates offline without the need of starting a new Flink job from scratch. For example, Flink users will be able to uncover topology or schema incompatibilities upon upgrading a Flink job, without having to load the state back to a running Flink job in the first place. Additionally, with upgradability dry runs [...]
+
+With all the exciting new functionality added in Flink 1.9 as well as some solid ideas and discussions around bringing state in Flink to the next level, the community is committed to making state in Apache Flink a fundamental element of the framework, something that is ever-present across versions and upgrades of your application and a component that is a true first-class citizen in Apache Flink. We encourage you to sign up to the [mailing list](https://flink.apache.org/community.html) [...]
diff --git a/img/blog/2020-01-29-state-unlocked-interacting-with-state-in-apache-flink/managing-state-in-flink-state-processor-api-visual-1.png b/img/blog/2020-01-29-state-unlocked-interacting-with-state-in-apache-flink/managing-state-in-flink-state-processor-api-visual-1.png
new file mode 100755
index 0000000..3ec4b5f
Binary files /dev/null and b/img/blog/2020-01-29-state-unlocked-interacting-with-state-in-apache-flink/managing-state-in-flink-state-processor-api-visual-1.png differ
diff --git a/img/blog/2020-01-29-state-unlocked-interacting-with-state-in-apache-flink/managing-state-in-flink-state-processor-api-visual-2.png b/img/blog/2020-01-29-state-unlocked-interacting-with-state-in-apache-flink/managing-state-in-flink-state-processor-api-visual-2.png
new file mode 100755
index 0000000..c5bd780
Binary files /dev/null and b/img/blog/2020-01-29-state-unlocked-interacting-with-state-in-apache-flink/managing-state-in-flink-state-processor-api-visual-2.png differ
diff --git a/img/blog/2020-01-29-state-unlocked-interacting-with-state-in-apache-flink/managing-state-in-flink-visual-1.png b/img/blog/2020-01-29-state-unlocked-interacting-with-state-in-apache-flink/managing-state-in-flink-visual-1.png
new file mode 100755
index 0000000..d78f128
Binary files /dev/null and b/img/blog/2020-01-29-state-unlocked-interacting-with-state-in-apache-flink/managing-state-in-flink-visual-1.png differ
diff --git a/img/blog/2020-01-29-state-unlocked-interacting-with-state-in-apache-flink/managing-state-in-flink-visual-2.png b/img/blog/2020-01-29-state-unlocked-interacting-with-state-in-apache-flink/managing-state-in-flink-visual-2.png
new file mode 100755
index 0000000..6ea5c78
Binary files /dev/null and b/img/blog/2020-01-29-state-unlocked-interacting-with-state-in-apache-flink/managing-state-in-flink-visual-2.png differ