You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by rm...@apache.org on 2021/01/18 11:31:23 UTC
[flink-web] branch asf-site updated (38b3a48 -> f255542)
This is an automated email from the ASF dual-hosted git repository.
rmetzger pushed a change to branch asf-site
in repository https://gitbox.apache.org/repos/asf/flink-web.git.
from 38b3a48 Rebuild page
new 50d2173 Add the rocksdb blog
new 5e0487e spill to disks -> spill to disk
new 635c74a cloud based -> cloud-based
new 1edd72b disks -> disk
new ab8d20d Cover also Flink 1.12
new 1bf7987 to enable -> enabling
new 031bf5a the JVM heap
new 185bb1a change date to 2021-01-18
new e93a79d rename file/dir to 2021-01-18
new f255542 Rebuild site
The 10 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
_posts/2021-01-18-rocksdb.md | 121 +++++++
content/2021/01/18/rocksdb.html | 369 +++++++++++++++++++++
content/blog/index.html | 42 +--
content/blog/page10/index.html | 42 ++-
content/blog/page11/index.html | 40 ++-
content/blog/page12/index.html | 40 ++-
content/blog/page13/index.html | 40 ++-
content/blog/page14/index.html | 40 ++-
content/blog/page15/index.html | 25 ++
content/blog/page2/index.html | 42 ++-
content/blog/page3/index.html | 38 ++-
content/blog/page4/index.html | 38 ++-
content/blog/page5/index.html | 36 +-
content/blog/page6/index.html | 36 +-
content/blog/page7/index.html | 36 +-
content/blog/page8/index.html | 38 ++-
content/blog/page9/index.html | 42 ++-
.../blog/2021-01-18-rocksdb/RocksDB-in-Flink.png | Bin 0 -> 72641 bytes
content/index.html | 6 +-
content/zh/index.html | 6 +-
img/blog/2021-01-18-rocksdb/RocksDB-in-Flink.png | Bin 0 -> 72641 bytes
21 files changed, 865 insertions(+), 212 deletions(-)
create mode 100644 _posts/2021-01-18-rocksdb.md
create mode 100644 content/2021/01/18/rocksdb.html
create mode 100644 content/img/blog/2021-01-18-rocksdb/RocksDB-in-Flink.png
create mode 100644 img/blog/2021-01-18-rocksdb/RocksDB-in-Flink.png
[flink-web] 07/10: the JVM heap
Posted by rm...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
rmetzger pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/flink-web.git
commit 031bf5acdcd4a08b93716f1174faa878004cee58
Author: qinjunjerry <11...@users.noreply.github.com>
AuthorDate: Tue Jan 12 14:26:42 2021 +0100
the JVM heap
Co-authored-by: morsapaes <ma...@gmail.com>
---
_posts/2020-12-20-rocksdb.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/_posts/2020-12-20-rocksdb.md b/_posts/2020-12-20-rocksdb.md
index 17e26d6..071a9c0 100644
--- a/_posts/2020-12-20-rocksdb.md
+++ b/_posts/2020-12-20-rocksdb.md
@@ -47,7 +47,7 @@ The command is for Linux only. For other operating systems, please refer to thei
In addition to RocksDBStateBackend, Flink has two other built-in state backends: MemoryStateBackend and FsStateBackend. They both are heap-based, as in-flight state is stored in the JVM heap. For the moment being, let’s ignore MemoryStateBackend, as it is intended only for **local developments** and **debugging**, not for production use.
-With RocksDBStateBackend, in-flight state is first written into off-heap/native memory, and then flushed to local disks when a configured threshold is reached. This means that RocksDBStateBackend can support state larger than the total configured heap capacity. The amount of state that you can store in RocksDBStateBackend is only limited by the amount of **disk space** available across the entire cluster. In addition, since RocksDBStateBackend doesn’t use JVM heap to store in-flight sta [...]
+With RocksDBStateBackend, in-flight state is first written into off-heap/native memory, and then flushed to local disks when a configured threshold is reached. This means that RocksDBStateBackend can support state larger than the total configured heap capacity. The amount of state that you can store in RocksDBStateBackend is only limited by the amount of **disk space** available across the entire cluster. In addition, since RocksDBStateBackend doesn’t use the JVM heap to store in-flight [...]
On top of full, self-contained state snapshots, RocksDBStateBackend also supports [incremental checkpointing](https://ci.apache.org/projects/flink/flink-docs-stable/ops/state/large_state_tuning.html#incremental-checkpoints) as a performance tuning option. An incremental checkpoint stores only the changes that occurred since the latest completed checkpoint. This dramatically reduces checkpointing time in comparison to performing a full snapshot. RocksDBStateBackend is currently the only s [...]
[flink-web] 04/10: disks -> disk
Posted by rm...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
rmetzger pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/flink-web.git
commit 1edd72bc83b9711bb8166720a870c52b73ae56a5
Author: qinjunjerry <11...@users.noreply.github.com>
AuthorDate: Tue Jan 12 13:56:55 2021 +0100
disks -> disk
Co-authored-by: morsapaes <ma...@gmail.com>
---
_posts/2020-12-20-rocksdb.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/_posts/2020-12-20-rocksdb.md b/_posts/2020-12-20-rocksdb.md
index 2d74e14..2252086 100644
--- a/_posts/2020-12-20-rocksdb.md
+++ b/_posts/2020-12-20-rocksdb.md
@@ -90,7 +90,7 @@ We hope this overview helped you gain a better understanding of the role of Rock
## State Location in RocksDB
-As mentioned earlier, in-flight state in RocksDBStateBackend is spilled to files on disks. These files are located under the directory specified by the Flink configuration [`state.backend.rocksdb.localdir`](https://ci.apache.org/projects/flink/flink-docs-stable/deployment/config.html#state-backend-rocksdb-localdir). Because disk performance has a direct impact on RocksDB’s performance, it’s recommended that this directory is located on a **local** disk. It’s discouraged to configure it [...]
+As mentioned earlier, in-flight state in RocksDBStateBackend is spilled to files on disk. These files are located under the directory specified by the Flink configuration [`state.backend.rocksdb.localdir`](https://ci.apache.org/projects/flink/flink-docs-stable/deployment/config.html#state-backend-rocksdb-localdir). Because disk performance has a direct impact on RocksDB’s performance, it’s recommended that this directory is located on a **local** disk. It’s discouraged to configure it t [...]
State snapshots are persisted to remote durable storage. During state snapshotting, TaskManagers take a snapshot of the in-flight state and store it remotely. Transferring the state snapshot to remote storage is handled purely by the TaskManager itself without the involvement of the state backend. So, [`state.checkpoints.dir`](https://ci.apache.org/projects/flink/flink-docs-stable/deployment/config.html#state-checkpoints-dir) or the parameter you set in the code for a particular job can [...]
[flink-web] 10/10: Rebuild site
Posted by rm...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
rmetzger pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/flink-web.git
commit f25554268003e1eca2b541fd9cb59465407b818a
Author: Robert Metzger <rm...@apache.org>
AuthorDate: Mon Jan 18 12:30:43 2021 +0100
Rebuild site
This closes #402
---
content/2021/01/18/rocksdb.html | 369 +++++++++++++++++++++
content/blog/index.html | 42 +--
content/blog/page10/index.html | 42 ++-
content/blog/page11/index.html | 40 ++-
content/blog/page12/index.html | 40 ++-
content/blog/page13/index.html | 40 ++-
content/blog/page14/index.html | 40 ++-
content/blog/page15/index.html | 25 ++
content/blog/page2/index.html | 42 ++-
content/blog/page3/index.html | 38 ++-
content/blog/page4/index.html | 38 ++-
content/blog/page5/index.html | 36 +-
content/blog/page6/index.html | 36 +-
content/blog/page7/index.html | 36 +-
content/blog/page8/index.html | 38 ++-
content/blog/page9/index.html | 42 ++-
.../blog/2021-01-18-rocksdb/RocksDB-in-Flink.png | Bin 0 -> 72641 bytes
content/index.html | 6 +-
content/zh/index.html | 6 +-
19 files changed, 744 insertions(+), 212 deletions(-)
diff --git a/content/2021/01/18/rocksdb.html b/content/2021/01/18/rocksdb.html
new file mode 100644
index 0000000..a3f8b7d
--- /dev/null
+++ b/content/2021/01/18/rocksdb.html
@@ -0,0 +1,369 @@
+<!DOCTYPE html>
+<html lang="en">
+ <head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+ <!-- The above 3 meta tags *must* come first in the head; any other head content must come *after* these tags -->
+ <title>Apache Flink: Using RocksDB State Backend in Apache Flink: When and How</title>
+ <link rel="shortcut icon" href="/favicon.ico" type="image/x-icon">
+ <link rel="icon" href="/favicon.ico" type="image/x-icon">
+
+ <!-- Bootstrap -->
+ <link rel="stylesheet" href="/css/bootstrap.min.css">
+ <link rel="stylesheet" href="/css/flink.css">
+ <link rel="stylesheet" href="/css/syntax.css">
+
+ <!-- Blog RSS feed -->
+ <link href="/blog/feed.xml" rel="alternate" type="application/rss+xml" title="Apache Flink Blog: RSS feed" />
+
+ <!-- jQuery (necessary for Bootstrap's JavaScript plugins) -->
+ <!-- We need to load Jquery in the header for custom google analytics event tracking-->
+ <script src="/js/jquery.min.js"></script>
+
+ <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries -->
+ <!-- WARNING: Respond.js doesn't work if you view the page via file:// -->
+ <!--[if lt IE 9]>
+ <script src="https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js"></script>
+ <script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js"></script>
+ <![endif]-->
+ </head>
+ <body>
+
+
+ <!-- Main content. -->
+ <div class="container">
+ <div class="row">
+
+
+ <div id="sidebar" class="col-sm-3">
+
+
+<!-- Top navbar. -->
+ <nav class="navbar navbar-default">
+ <!-- The logo. -->
+ <div class="navbar-header">
+ <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#bs-example-navbar-collapse-1">
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+ <div class="navbar-logo">
+ <a href="/">
+ <img alt="Apache Flink" src="/img/flink-header-logo.svg" width="147px" height="73px">
+ </a>
+ </div>
+ </div><!-- /.navbar-header -->
+
+ <!-- The navigation links. -->
+ <div class="collapse navbar-collapse" id="bs-example-navbar-collapse-1">
+ <ul class="nav navbar-nav navbar-main">
+
+ <!-- First menu section explains visitors what Flink is -->
+
+ <!-- What is Stream Processing? -->
+ <!--
+ <li><a href="/streamprocessing1.html">What is Stream Processing?</a></li>
+ -->
+
+ <!-- What is Flink? -->
+ <li><a href="/flink-architecture.html">What is Apache Flink?</a></li>
+
+
+
+ <!-- What is Stateful Functions? -->
+
+ <li><a href="/stateful-functions.html">What is Stateful Functions?</a></li>
+
+ <!-- Use cases -->
+ <li><a href="/usecases.html">Use Cases</a></li>
+
+ <!-- Powered by -->
+ <li><a href="/poweredby.html">Powered By</a></li>
+
+
+
+ <!-- Second menu section aims to support Flink users -->
+
+ <!-- Downloads -->
+ <li><a href="/downloads.html">Downloads</a></li>
+
+ <!-- Getting Started -->
+ <li class="dropdown">
+ <a class="dropdown-toggle" data-toggle="dropdown" href="#">Getting Started<span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li><a href="https://ci.apache.org/projects/flink/flink-docs-release-1.12/try-flink/index.html" target="_blank">With Flink <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+ <li><a href="https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.2/getting-started/project-setup.html" target="_blank">With Flink Stateful Functions <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+ <li><a href="/training.html">Training Course</a></li>
+ </ul>
+ </li>
+
+ <!-- Documentation -->
+ <li class="dropdown">
+ <a class="dropdown-toggle" data-toggle="dropdown" href="#">Documentation<span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li><a href="https://ci.apache.org/projects/flink/flink-docs-release-1.12" target="_blank">Flink 1.12 (Latest stable release) <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+ <li><a href="https://ci.apache.org/projects/flink/flink-docs-master" target="_blank">Flink Master (Latest Snapshot) <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+ <li><a href="https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.2" target="_blank">Flink Stateful Functions 2.2 (Latest stable release) <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+ <li><a href="https://ci.apache.org/projects/flink/flink-statefun-docs-master" target="_blank">Flink Stateful Functions Master (Latest Snapshot) <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+ </ul>
+ </li>
+
+ <!-- getting help -->
+ <li><a href="/gettinghelp.html">Getting Help</a></li>
+
+ <!-- Blog -->
+ <li><a href="/blog/"><b>Flink Blog</b></a></li>
+
+
+ <!-- Flink-packages -->
+ <li>
+ <a href="https://flink-packages.org" target="_blank">flink-packages.org <small><span class="glyphicon glyphicon-new-window"></span></small></a>
+ </li>
+
+
+ <!-- Third menu section aim to support community and contributors -->
+
+ <!-- Community -->
+ <li><a href="/community.html">Community & Project Info</a></li>
+
+ <!-- Roadmap -->
+ <li><a href="/roadmap.html">Roadmap</a></li>
+
+ <!-- Contribute -->
+ <li><a href="/contributing/how-to-contribute.html">How to Contribute</a></li>
+
+
+ <!-- GitHub -->
+ <li>
+ <a href="https://github.com/apache/flink" target="_blank">Flink on GitHub <small><span class="glyphicon glyphicon-new-window"></span></small></a>
+ </li>
+
+
+
+ <!-- Language Switcher -->
+ <li>
+
+
+ <a href="/zh/2021/01/18/rocksdb.html">中文版</a>
+
+
+ </li>
+
+ </ul>
+
+ <style>
+ .smalllinks:link {
+ display: inline-block !important; background: none; padding-top: 0px; padding-bottom: 0px; padding-right: 0px; min-width: 75px;
+ }
+ </style>
+
+ <ul class="nav navbar-nav navbar-bottom">
+ <hr />
+
+ <!-- Twitter -->
+ <li><a href="https://twitter.com/apacheflink" target="_blank">@ApacheFlink <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+
+ <!-- Visualizer -->
+ <li class=" hidden-md hidden-sm"><a href="/visualizer/" target="_blank">Plan Visualizer <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+
+ <li >
+ <a href="/security.html">Flink Security</a>
+ </li>
+
+ <hr />
+
+ <li><a href="https://apache.org" target="_blank">Apache Software Foundation <small><span class="glyphicon glyphicon-new-window"></span></small></a></li>
+
+ <li>
+
+ <a class="smalllinks" href="https://www.apache.org/licenses/" target="_blank">License</a> <small><span class="glyphicon glyphicon-new-window"></span></small>
+
+ <a class="smalllinks" href="https://www.apache.org/security/" target="_blank">Security</a> <small><span class="glyphicon glyphicon-new-window"></span></small>
+
+ <a class="smalllinks" href="https://www.apache.org/foundation/sponsorship.html" target="_blank">Donate</a> <small><span class="glyphicon glyphicon-new-window"></span></small>
+
+ <a class="smalllinks" href="https://www.apache.org/foundation/thanks.html" target="_blank">Thanks</a> <small><span class="glyphicon glyphicon-new-window"></span></small>
+ </li>
+
+ </ul>
+ </div><!-- /.navbar-collapse -->
+ </nav>
+
+ </div>
+ <div class="col-sm-9">
+ <div class="row-fluid">
+ <div class="col-sm-12">
+ <div class="row">
+ <h1>Using RocksDB State Backend in Apache Flink: When and How</h1>
+ <p><i></i></p>
+
+ <article>
+ <p>18 Jan 2021 Jun Qin </p>
+
+<p>Stream processing applications are often stateful, “remembering” information from processed events and using it to influence further event processing. In Flink, the remembered information, i.e., state, is stored locally in the configured state backend. To prevent data loss in case of failures, the state backend periodically persists a snapshot of its contents to a pre-configured durable storage. The <a href="https://rocksdb.org/">RocksDB</a> state backend (i.e., RocksDBStateBackend) i [...]
+
+<h1 id="state-in-flink">State in Flink</h1>
+
+<p>To best understand state and state backends in Flink, it’s important to distinguish between <strong>in-flight state</strong> and <strong>state snapshots</strong>. In-flight state, also known as working state, is the state a Flink job is working on. It is always stored locally in memory (with the possibility to spill to disk) and can be lost when jobs fail without impacting job recoverability. State snapshots, i.e., <a href="https://ci.apache.org/projects/flink/flink-docs-stable/ops/st [...]
+
+<h1 id="what-is-rocksdb">What is RocksDB?</h1>
+
+<p>Thinking of RocksDB as a distributed database that needs to run on a cluster and to be managed by specialized administrators is a common misconception. RocksDB is an embeddable persistent key-value store for fast storage. It interacts with Flink via the Java Native Interface (JNI). The picture below shows where RocksDB fits in a Flink cluster node. Details are explained in the following sections.</p>
+
+<center>
+<img vspace="8" style="width:60%" src="/img/blog/2021-01-18-rocksdb/RocksDB-in-Flink.png" />
+</center>
+
+<h1 id="rocksdb-in-flink">RocksDB in Flink</h1>
+
+<p>Everything you need to use RocksDB as a state backend is bundled in the Apache Flink distribution, including the native shared library:</p>
+
+<div class="highlight"><pre><code>$ jar -tvf lib/flink-dist_2.12-1.12.0.jar| grep librocksdbjni-linux64
+8695334 Wed Nov 27 02:27:06 CET 2019 librocksdbjni-linux64.so
+</code></pre></div>
+
+<p>At runtime, RocksDB is embedded in the TaskManager processes. It runs in native threads and works with local files. For example, if you have a job configured with RocksDBStateBackend running in your Flink cluster, you’ll see something similar to the following, where 32513 is the TaskManager process ID.</p>
+
+<div class="highlight"><pre><code>$ ps -T -p 32513 | grep -i rocksdb
+32513 32633 ? 00:00:00 rocksdb:low0
+32513 32634 ? 00:00:00 rocksdb:high0
+</code></pre></div>
+
+<div class="alert alert-info">
+ <p><span class="label label-info" style="display: inline-block"><span class="glyphicon glyphicon-info-sign" aria-hidden="true"></span> Note</span>
+The command is for Linux only. For other operating systems, please refer to their documentation.</p>
+</div>
+
+<h1 id="when-to-use-rocksdbstatebackend">When to use RocksDBStateBackend</h1>
+
+<p>In addition to RocksDBStateBackend, Flink has two other built-in state backends: MemoryStateBackend and FsStateBackend. They both are heap-based, as in-flight state is stored in the JVM heap. For the moment being, let’s ignore MemoryStateBackend, as it is intended only for <strong>local developments</strong> and <strong>debugging</strong>, not for production use.</p>
+
+<p>With RocksDBStateBackend, in-flight state is first written into off-heap/native memory, and then flushed to local disks when a configured threshold is reached. This means that RocksDBStateBackend can support state larger than the total configured heap capacity. The amount of state that you can store in RocksDBStateBackend is only limited by the amount of <strong>disk space</strong> available across the entire cluster. In addition, since RocksDBStateBackend doesn’t use the JVM heap to [...]
+
+<p>On top of full, self-contained state snapshots, RocksDBStateBackend also supports <a href="https://ci.apache.org/projects/flink/flink-docs-stable/ops/state/large_state_tuning.html#incremental-checkpoints">incremental checkpointing</a> as a performance tuning option. An incremental checkpoint stores only the changes that occurred since the latest completed checkpoint. This dramatically reduces checkpointing time in comparison to performing a full snapshot. RocksDBStateBackend is curren [...]
+
+<p>RocksDB is a good option when:</p>
+
+<ul>
+ <li>The state of your job is larger than can fit in local memory (e.g., long windows, large <a href="https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/state.html">keyed state</a>);</li>
+ <li>You’re looking into incremental checkpointing as a way to reduce checkpointing time;</li>
+ <li>You expect to have more predictable latency without being impacted by JVM Garbage Collection.</li>
+</ul>
+
+<p>Otherwise, if your application has <strong>small state</strong> or requires <strong>very low latency</strong>, you should consider <strong>FsStateBackend</strong>. As a rule of thumb, RocksDBStateBackend is a few times slower than heap-based state backends, because it stores key/value pairs as serialized bytes. This means that any state access (reads/writes) needs to go through a de-/serialization process crossing the JNI boundary, which is more expensive than working directly with th [...]
+
+<h1 id="how-to-use-rocksdbstatebackend">How to use RocksDBStateBackend</h1>
+
+<p>RocksDB is fully embedded within and fully managed by the TaskManager process. RocksDBStateBackend can be configured at the cluster level as the default for the entire cluster, or at the job level for individual jobs. The job level configuration takes precedence over the cluster level configuration.</p>
+
+<h2 id="cluster-level">Cluster Level</h2>
+
+<p>Add the following configuration in <a href="https://ci.apache.org/projects/flink/flink-docs-stable/deployment/config.html"><code>conf/flink-conf.yaml</code></a>:</p>
+
+<div class="highlight"><pre><code>state.backend: rocksdb
+state.backend.incremental: true
+state.checkpoints.dir: hdfs:///flink-checkpoints # location to store checkpoints
+</code></pre></div>
+
+<h1 id="job-level">Job Level</h1>
+
+<p>Add the following into your job’s code after StreamExecutionEnvironment is created:</p>
+
+<div class="highlight"><pre><code># 'env' is the created StreamExecutionEnvironment
+# 'true' is to enable incremental checkpointing
+env.setStateBackend(new RocksDBStateBackend("hdfs:///fink-checkpoints", true));
+</code></pre></div>
+
+<div class="alert alert-info">
+ <p><span class="label label-info" style="display: inline-block"><span class="glyphicon glyphicon-info-sign" aria-hidden="true"></span> Note</span>
+In addition to HDFS, you can also use other on-premises or cloud-based object stores if the corresponding dependencies are added under <a href="https://ci.apache.org/projects/flink/flink-docs-stable/deployment/filesystems/plugins.html">FLINK_HOME/plugins</a>.</p>
+</div>
+
+<h1 id="best-practices-and-advanced-configuration">Best Practices and Advanced Configuration</h1>
+
+<p>We hope this overview helped you gain a better understanding of the role of RocksDB in Flink and how to successfully run a job with RocksDBStateBackend. To round it off, we’ll explore some best practices and a few reference points for further troubleshooting and performance tuning.</p>
+
+<h2 id="state-location-in-rocksdb">State Location in RocksDB</h2>
+
+<p>As mentioned earlier, in-flight state in RocksDBStateBackend is spilled to files on disk. These files are located under the directory specified by the Flink configuration <a href="https://ci.apache.org/projects/flink/flink-docs-stable/deployment/config.html#state-backend-rocksdb-localdir"><code>state.backend.rocksdb.localdir</code></a>. Because disk performance has a direct impact on RocksDB’s performance, it’s recommended that this directory is located on a <strong>local</strong> dis [...]
+
+<p>State snapshots are persisted to remote durable storage. During state snapshotting, TaskManagers take a snapshot of the in-flight state and store it remotely. Transferring the state snapshot to remote storage is handled purely by the TaskManager itself without the involvement of the state backend. So, <a href="https://ci.apache.org/projects/flink/flink-docs-stable/deployment/config.html#state-checkpoints-dir"><code>state.checkpoints.dir</code></a> or the parameter you set in the code [...]
+
+<h2 id="troubleshooting-rocksdb">Troubleshooting RocksDB</h2>
+
+<p>To check how RocksDB is behaving in production, you should look for the RocksDB log file named LOG. By default, this log file is located in the same directory as your data files, i.e., the directory specified by the Flink configuration <a href="https://ci.apache.org/projects/flink/flink-docs-stable/deployment/config.html#state-backend-rocksdb-localdir"><code>state.backend.rocksdb.localdir</code></a>. When enabled, <a href="https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide# [...]
+
+<div class="alert alert-info">
+ <p><span class="label label-info" style="display: inline-block"><span class="glyphicon glyphicon-info-sign" aria-hidden="true"></span> Note</span>
+From Flink 1.10, RocksDB logging was effectively disabled by <a href="https://github.com/apache/flink/blob/master/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/PredefinedOptions.java#L64">setting the log level to HEADER</a>. To enable it, check <a href="https://ververica.zendesk.com/hc/en-us/articles/360015933320-How-to-get-RocksDB-s-LOG-file-back-for-advanced-troubleshooting">How to get RocksDB’s LOG file back for advanced trouble [...]
+</div>
+
+<div class="alert alert-warning">
+ <p><span class="label label-warning" style="display: inline-block"><span class="glyphicon glyphicon-warning-sign" aria-hidden="true"></span> Warning</span>
+Enabling RocksDB’s native metrics in Flink may have a negative performance impact on your job.</p>
+</div>
+
+<h2 id="tuning-rocksdb">Tuning RocksDB</h2>
+
+<p>Since Flink 1.10, Flink configures RocksDB’s memory allocation to the amount of managed memory of each task slot by default. The primary mechanism for improving memory-related performance issues is to increase Flink’s <a href="https://ci.apache.org/projects/flink/flink-docs-stable/deployment/memory/mem_setup_tm.html#managed-memory">managed memory</a> via the Flink configuration <a href="https://ci.apache.org/projects/flink/flink-docs-stable/deployment/config.html#taskmanager-memory-ma [...]
+
+<p>While data is being written or overwritten in RocksDB, flushing from memory to local disks and data compaction are managed in the background by RocksDB threads. On a machine with many CPU cores, you should increase the parallelism of background flushing and compaction by setting the Flink configuration <a href="https://ci.apache.org/projects/flink/flink-docs-stable/deployment/config.html#state-backend-rocksdb-thread-num"><code>state.backend.rocksdb.thread.num</code></a> (corresponding [...]
+
+<p>For other RocksDBStateBackend configurations, check the Flink documentation on <a href="https://ci.apache.org/projects/flink/flink-docs-stable/deployment/config.html#advanced-rocksdb-state-backends-options">Advanced RocksDB State Backends Options</a>. For further tuning, check <a href="https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide">RocksDB Tuning Guide</a> in <a href="https://github.com/facebook/rocksdb/wiki">RocksDB Wiki</a>.</p>
+
+<h1 id="conclusion">Conclusion</h1>
+
+<p>The <a href="https://rocksdb.org/">RocksDB</a> state backend (i.e., RocksDBStateBackend) is one of the three state backends bundled in Flink, and can be a powerful choice when configuring your streaming applications. It enables scalable applications maintaining up to many terabytes of state with exactly-once processing guarantees. If the state of your Flink job is too large to fit on the JVM heap, you are interested in incremental checkpointing, or you expect to have predictable late [...]
+
+ </article>
+ </div>
+
+ <div class="row">
+ <div id="disqus_thread"></div>
+ <script type="text/javascript">
+ /* * * CONFIGURATION VARIABLES: EDIT BEFORE PASTING INTO YOUR WEBPAGE * * */
+ var disqus_shortname = 'stratosphere-eu'; // required: replace example with your forum shortname
+
+ /* * * DON'T EDIT BELOW THIS LINE * * */
+ (function() {
+ var dsq = document.createElement('script'); dsq.type = 'text/javascript'; dsq.async = true;
+ dsq.src = '//' + disqus_shortname + '.disqus.com/embed.js';
+ (document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(dsq);
+ })();
+ </script>
+ </div>
+ </div>
+</div>
+ </div>
+ </div>
+
+ <hr />
+
+ <div class="row">
+ <div class="footer text-center col-sm-12">
+ <p>Copyright © 2014-2019 <a href="http://apache.org">The Apache Software Foundation</a>. All Rights Reserved.</p>
+ <p>Apache Flink, Flink®, Apache®, the squirrel logo, and the Apache feather logo are either registered trademarks or trademarks of The Apache Software Foundation.</p>
+ <p><a href="/privacy-policy.html">Privacy Policy</a> · <a href="/blog/feed.xml">RSS feed</a></p>
+ </div>
+ </div>
+ </div><!-- /.container -->
+
+ <!-- Include all compiled plugins (below), or include individual files as needed -->
+ <script src="/js/jquery.matchHeight-min.js"></script>
+ <script src="/js/bootstrap.min.js"></script>
+ <script src="/js/codetabs.js"></script>
+ <script src="/js/stickysidebar.js"></script>
+
+ <!-- Google Analytics -->
+ <script>
+ (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
+ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
+ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
+ })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
+
+ ga('create', 'UA-52545728-1', 'auto');
+ ga('send', 'pageview');
+ </script>
+ </body>
+</html>
diff --git a/content/blog/index.html b/content/blog/index.html
index 3af0370..a9bdbde 100644
--- a/content/blog/index.html
+++ b/content/blog/index.html
@@ -201,6 +201,19 @@
<!-- Blog posts -->
<article>
+ <h2 class="blog-title"><a href="/2021/01/18/rocksdb.html">Using RocksDB State Backend in Apache Flink: When and How</a></h2>
+
+ <p>18 Jan 2021
+ Jun Qin </p>
+
+ <p>This blog post will guide you through the benefits of using RocksDB to manage your application’s state, explain when and how to use it and also clear up a few common misconceptions.</p>
+
+ <p><a href="/2021/01/18/rocksdb.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
<h2 class="blog-title"><a href="/news/2021/01/11/batch-fine-grained-fault-tolerance.html">Exploring fine-grained recovery of bounded data sets on Flink</a></h2>
<p>11 Jan 2021
@@ -323,25 +336,6 @@
<hr>
- <article>
- <h2 class="blog-title"><a href="/news/2020/09/28/release-statefun-2.2.0.html">Stateful Functions 2.2.0 Release Announcement</a></h2>
-
- <p>28 Sep 2020
- Tzu-Li (Gordon) Tai (<a href="https://twitter.com/tzulitai">@tzulitai</a>) & Igal Shilman (<a href="https://twitter.com/IgalShilman">@IgalShilman</a>)</p>
-
- <p><p>The Apache Flink community is happy to announce the release of Stateful Functions (StateFun) 2.2.0! This release
-introduces major features that extend the SDKs, such as support for asynchronous functions in the Python SDK, new
-persisted state constructs, and a new SDK that allows embedding StateFun functions within a Flink DataStream job.
-Moreover, we’ve also included important changes that improve out-of-the-box stability for common workloads,
-as well as increased observability for operational purposes.</p>
-
-</p>
-
- <p><a href="/news/2020/09/28/release-statefun-2.2.0.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -374,6 +368,16 @@ as well as increased observability for operational purposes.</p>
<ul id="markdown-toc">
+ <li><a href="/2021/01/18/rocksdb.html">Using RocksDB State Backend in Apache Flink: When and How</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/news/2021/01/11/batch-fine-grained-fault-tolerance.html">Exploring fine-grained recovery of bounded data sets on Flink</a></li>
diff --git a/content/blog/page10/index.html b/content/blog/page10/index.html
index 207c4be..a486d05 100644
--- a/content/blog/page10/index.html
+++ b/content/blog/page10/index.html
@@ -201,6 +201,23 @@
<!-- Blog posts -->
<article>
+ <h2 class="blog-title"><a href="/news/2017/12/12/release-1.4.0.html">Apache Flink 1.4.0 Release Announcement</a></h2>
+
+ <p>12 Dec 2017
+ Aljoscha Krettek (<a href="https://twitter.com/aljoscha">@aljoscha</a>) & Mike Winters (<a href="https://twitter.com/wints">@wints</a>)</p>
+
+ <p><p>The Apache Flink community is pleased to announce the 1.4.0 release. Over the past 5 months, the
+Flink community has been working hard to resolve more than 900 issues. See the <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12340533">complete changelog</a>
+for more detail.</p>
+
+</p>
+
+ <p><a href="/news/2017/12/12/release-1.4.0.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
<h2 class="blog-title"><a href="/news/2017/11/22/release-1.4-and-1.5-timeline.html">Looking Ahead to Apache Flink 1.4.0 and 1.5.0</a></h2>
<p>22 Nov 2017
@@ -333,21 +350,6 @@ what’s coming in Flink 1.4.0 as well as a preview of what the Flink community
<hr>
- <article>
- <h2 class="blog-title"><a href="/news/2017/03/23/release-1.1.5.html">Apache Flink 1.1.5 Released</a></h2>
-
- <p>23 Mar 2017
- </p>
-
- <p><p>The Apache Flink community released the next bugfix version of the Apache Flink 1.1 series.</p>
-
-</p>
-
- <p><a href="/news/2017/03/23/release-1.1.5.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -380,6 +382,16 @@ what’s coming in Flink 1.4.0 as well as a preview of what the Flink community
<ul id="markdown-toc">
+ <li><a href="/2021/01/18/rocksdb.html">Using RocksDB State Backend in Apache Flink: When and How</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/news/2021/01/11/batch-fine-grained-fault-tolerance.html">Exploring fine-grained recovery of bounded data sets on Flink</a></li>
diff --git a/content/blog/page11/index.html b/content/blog/page11/index.html
index 170f525..2341804 100644
--- a/content/blog/page11/index.html
+++ b/content/blog/page11/index.html
@@ -201,6 +201,21 @@
<!-- Blog posts -->
<article>
+ <h2 class="blog-title"><a href="/news/2017/03/23/release-1.1.5.html">Apache Flink 1.1.5 Released</a></h2>
+
+ <p>23 Mar 2017
+ </p>
+
+ <p><p>The Apache Flink community released the next bugfix version of the Apache Flink 1.1 series.</p>
+
+</p>
+
+ <p><a href="/news/2017/03/23/release-1.1.5.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
<h2 class="blog-title"><a href="/news/2017/02/06/release-1.2.0.html">Announcing Apache Flink 1.2.0</a></h2>
<p>06 Feb 2017 by Robert Metzger
@@ -330,21 +345,6 @@
<hr>
- <article>
- <h2 class="blog-title"><a href="/news/2016/05/11/release-1.0.3.html">Flink 1.0.3 Released</a></h2>
-
- <p>11 May 2016
- </p>
-
- <p><p>Today, the Flink community released Flink version <strong>1.0.3</strong>, the third bugfix release of the 1.0 series.</p>
-
-</p>
-
- <p><a href="/news/2016/05/11/release-1.0.3.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -377,6 +377,16 @@
<ul id="markdown-toc">
+ <li><a href="/2021/01/18/rocksdb.html">Using RocksDB State Backend in Apache Flink: When and How</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/news/2021/01/11/batch-fine-grained-fault-tolerance.html">Exploring fine-grained recovery of bounded data sets on Flink</a></li>
diff --git a/content/blog/page12/index.html b/content/blog/page12/index.html
index 777854e..02d3b49 100644
--- a/content/blog/page12/index.html
+++ b/content/blog/page12/index.html
@@ -201,6 +201,21 @@
<!-- Blog posts -->
<article>
+ <h2 class="blog-title"><a href="/news/2016/05/11/release-1.0.3.html">Flink 1.0.3 Released</a></h2>
+
+ <p>11 May 2016
+ </p>
+
+ <p><p>Today, the Flink community released Flink version <strong>1.0.3</strong>, the third bugfix release of the 1.0 series.</p>
+
+</p>
+
+ <p><a href="/news/2016/05/11/release-1.0.3.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
<h2 class="blog-title"><a href="/news/2016/04/22/release-1.0.2.html">Flink 1.0.2 Released</a></h2>
<p>22 Apr 2016
@@ -328,21 +343,6 @@
<hr>
- <article>
- <h2 class="blog-title"><a href="/news/2015/11/27/release-0.10.1.html">Flink 0.10.1 released</a></h2>
-
- <p>27 Nov 2015
- </p>
-
- <p><p>Today, the Flink community released the first bugfix release of the 0.10 series of Flink.</p>
-
-</p>
-
- <p><a href="/news/2015/11/27/release-0.10.1.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -375,6 +375,16 @@
<ul id="markdown-toc">
+ <li><a href="/2021/01/18/rocksdb.html">Using RocksDB State Backend in Apache Flink: When and How</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/news/2021/01/11/batch-fine-grained-fault-tolerance.html">Exploring fine-grained recovery of bounded data sets on Flink</a></li>
diff --git a/content/blog/page13/index.html b/content/blog/page13/index.html
index 3a65f65..7550aec 100644
--- a/content/blog/page13/index.html
+++ b/content/blog/page13/index.html
@@ -201,6 +201,21 @@
<!-- Blog posts -->
<article>
+ <h2 class="blog-title"><a href="/news/2015/11/27/release-0.10.1.html">Flink 0.10.1 released</a></h2>
+
+ <p>27 Nov 2015
+ </p>
+
+ <p><p>Today, the Flink community released the first bugfix release of the 0.10 series of Flink.</p>
+
+</p>
+
+ <p><a href="/news/2015/11/27/release-0.10.1.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
<h2 class="blog-title"><a href="/news/2015/11/16/release-0.10.0.html">Announcing Apache Flink 0.10.0</a></h2>
<p>16 Nov 2015
@@ -343,21 +358,6 @@ release is a preview release that contains known issues.</p>
<hr>
- <article>
- <h2 class="blog-title"><a href="/news/2015/04/07/march-in-flink.html">March 2015 in the Flink community</a></h2>
-
- <p>07 Apr 2015
- </p>
-
- <p><p>March has been a busy month in the Flink community.</p>
-
-</p>
-
- <p><a href="/news/2015/04/07/march-in-flink.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -390,6 +390,16 @@ release is a preview release that contains known issues.</p>
<ul id="markdown-toc">
+ <li><a href="/2021/01/18/rocksdb.html">Using RocksDB State Backend in Apache Flink: When and How</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/news/2021/01/11/batch-fine-grained-fault-tolerance.html">Exploring fine-grained recovery of bounded data sets on Flink</a></li>
diff --git a/content/blog/page14/index.html b/content/blog/page14/index.html
index f6b7395..2f95413 100644
--- a/content/blog/page14/index.html
+++ b/content/blog/page14/index.html
@@ -201,6 +201,21 @@
<!-- Blog posts -->
<article>
+ <h2 class="blog-title"><a href="/news/2015/04/07/march-in-flink.html">March 2015 in the Flink community</a></h2>
+
+ <p>07 Apr 2015
+ </p>
+
+ <p><p>March has been a busy month in the Flink community.</p>
+
+</p>
+
+ <p><a href="/news/2015/04/07/march-in-flink.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
<h2 class="blog-title"><a href="/news/2015/03/13/peeking-into-Apache-Flinks-Engine-Room.html">Peeking into Apache Flink's Engine Room</a></h2>
<p>13 Mar 2015 by Fabian Hüske (<a href="https://twitter.com/">@fhueske</a>)
@@ -340,21 +355,6 @@ and offers a new API including definition of flexible windows.</p>
<hr>
- <article>
- <h2 class="blog-title"><a href="/news/2014/09/26/release-0.6.1.html">Apache Flink 0.6.1 available</a></h2>
-
- <p>26 Sep 2014
- </p>
-
- <p><p>We are happy to announce the availability of Flink 0.6.1.</p>
-
-</p>
-
- <p><a href="/news/2014/09/26/release-0.6.1.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -387,6 +387,16 @@ and offers a new API including definition of flexible windows.</p>
<ul id="markdown-toc">
+ <li><a href="/2021/01/18/rocksdb.html">Using RocksDB State Backend in Apache Flink: When and How</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/news/2021/01/11/batch-fine-grained-fault-tolerance.html">Exploring fine-grained recovery of bounded data sets on Flink</a></li>
diff --git a/content/blog/page15/index.html b/content/blog/page15/index.html
index 8153713..c33cea3 100644
--- a/content/blog/page15/index.html
+++ b/content/blog/page15/index.html
@@ -201,6 +201,21 @@
<!-- Blog posts -->
<article>
+ <h2 class="blog-title"><a href="/news/2014/09/26/release-0.6.1.html">Apache Flink 0.6.1 available</a></h2>
+
+ <p>26 Sep 2014
+ </p>
+
+ <p><p>We are happy to announce the availability of Flink 0.6.1.</p>
+
+</p>
+
+ <p><a href="/news/2014/09/26/release-0.6.1.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
<h2 class="blog-title"><a href="/news/2014/08/26/release-0.6.html">Apache Flink 0.6 available</a></h2>
<p>26 Aug 2014
@@ -250,6 +265,16 @@ academic and open source project that Flink originates from.</p>
<ul id="markdown-toc">
+ <li><a href="/2021/01/18/rocksdb.html">Using RocksDB State Backend in Apache Flink: When and How</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/news/2021/01/11/batch-fine-grained-fault-tolerance.html">Exploring fine-grained recovery of bounded data sets on Flink</a></li>
diff --git a/content/blog/page2/index.html b/content/blog/page2/index.html
index 2755aeb..8c78437 100644
--- a/content/blog/page2/index.html
+++ b/content/blog/page2/index.html
@@ -201,6 +201,25 @@
<!-- Blog posts -->
<article>
+ <h2 class="blog-title"><a href="/news/2020/09/28/release-statefun-2.2.0.html">Stateful Functions 2.2.0 Release Announcement</a></h2>
+
+ <p>28 Sep 2020
+ Tzu-Li (Gordon) Tai (<a href="https://twitter.com/tzulitai">@tzulitai</a>) & Igal Shilman (<a href="https://twitter.com/IgalShilman">@IgalShilman</a>)</p>
+
+ <p><p>The Apache Flink community is happy to announce the release of Stateful Functions (StateFun) 2.2.0! This release
+introduces major features that extend the SDKs, such as support for asynchronous functions in the Python SDK, new
+persisted state constructs, and a new SDK that allows embedding StateFun functions within a Flink DataStream job.
+Moreover, we’ve also included important changes that improve out-of-the-box stability for common workloads,
+as well as increased observability for operational purposes.</p>
+
+</p>
+
+ <p><a href="/news/2020/09/28/release-statefun-2.2.0.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
<h2 class="blog-title"><a href="/news/2020/09/17/release-1.11.2.html">Apache Flink 1.11.2 Released</a></h2>
<p>17 Sep 2020
@@ -323,19 +342,6 @@
<hr>
- <article>
- <h2 class="blog-title"><a href="/2020/07/28/flink-sql-demo-building-e2e-streaming-application.html">Flink SQL Demo: Building an End-to-End Streaming Application</a></h2>
-
- <p>28 Jul 2020
- Jark Wu (<a href="https://twitter.com/JarkWu">@JarkWu</a>)</p>
-
- <p>Apache Flink 1.11 has released many exciting new features, including many developments in Flink SQL which is evolving at a fast pace. This article takes a closer look at how to quickly build streaming applications with Flink SQL from a practical point of view.</p>
-
- <p><a href="/2020/07/28/flink-sql-demo-building-e2e-streaming-application.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -368,6 +374,16 @@
<ul id="markdown-toc">
+ <li><a href="/2021/01/18/rocksdb.html">Using RocksDB State Backend in Apache Flink: When and How</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/news/2021/01/11/batch-fine-grained-fault-tolerance.html">Exploring fine-grained recovery of bounded data sets on Flink</a></li>
diff --git a/content/blog/page3/index.html b/content/blog/page3/index.html
index fe7eefa..ba13f56 100644
--- a/content/blog/page3/index.html
+++ b/content/blog/page3/index.html
@@ -201,6 +201,19 @@
<!-- Blog posts -->
<article>
+ <h2 class="blog-title"><a href="/2020/07/28/flink-sql-demo-building-e2e-streaming-application.html">Flink SQL Demo: Building an End-to-End Streaming Application</a></h2>
+
+ <p>28 Jul 2020
+ Jark Wu (<a href="https://twitter.com/JarkWu">@JarkWu</a>)</p>
+
+ <p>Apache Flink 1.11 has released many exciting new features, including many developments in Flink SQL which is evolving at a fast pace. This article takes a closer look at how to quickly build streaming applications with Flink SQL from a practical point of view.</p>
+
+ <p><a href="/2020/07/28/flink-sql-demo-building-e2e-streaming-application.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
<h2 class="blog-title"><a href="/news/2020/07/27/community-update.html">Flink Community Update - July'20</a></h2>
<p>27 Jul 2020
@@ -337,21 +350,6 @@ and provide a tutorial for running Streaming ETL with Flink on Zeppelin.</p>
<hr>
- <article>
- <h2 class="blog-title"><a href="/news/2020/05/12/release-1.10.1.html">Apache Flink 1.10.1 Released</a></h2>
-
- <p>12 May 2020
- Yu Li (<a href="https://twitter.com/LiyuApache">@LiyuApache</a>)</p>
-
- <p><p>The Apache Flink community released the first bugfix version of the Apache Flink 1.10 series.</p>
-
-</p>
-
- <p><a href="/news/2020/05/12/release-1.10.1.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -384,6 +382,16 @@ and provide a tutorial for running Streaming ETL with Flink on Zeppelin.</p>
<ul id="markdown-toc">
+ <li><a href="/2021/01/18/rocksdb.html">Using RocksDB State Backend in Apache Flink: When and How</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/news/2021/01/11/batch-fine-grained-fault-tolerance.html">Exploring fine-grained recovery of bounded data sets on Flink</a></li>
diff --git a/content/blog/page4/index.html b/content/blog/page4/index.html
index f4d7b77..a10c19b 100644
--- a/content/blog/page4/index.html
+++ b/content/blog/page4/index.html
@@ -201,6 +201,21 @@
<!-- Blog posts -->
<article>
+ <h2 class="blog-title"><a href="/news/2020/05/12/release-1.10.1.html">Apache Flink 1.10.1 Released</a></h2>
+
+ <p>12 May 2020
+ Yu Li (<a href="https://twitter.com/LiyuApache">@LiyuApache</a>)</p>
+
+ <p><p>The Apache Flink community released the first bugfix version of the Apache Flink 1.10 series.</p>
+
+</p>
+
+ <p><a href="/news/2020/05/12/release-1.10.1.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
<h2 class="blog-title"><a href="/news/2020/05/07/community-update.html">Flink Community Update - May'20</a></h2>
<p>07 May 2020
@@ -324,19 +339,6 @@ This release marks a big milestone: Stateful Functions 2.0 is not only an API up
<hr>
- <article>
- <h2 class="blog-title"><a href="/news/2020/03/24/demo-fraud-detection-2.html">Advanced Flink Application Patterns Vol.2: Dynamic Updates of Application Logic</a></h2>
-
- <p>24 Mar 2020
- Alexander Fedulov (<a href="https://twitter.com/alex_fedulov">@alex_fedulov</a>)</p>
-
- <p>In this series of blog posts you will learn about powerful Flink patterns for building streaming applications.</p>
-
- <p><a href="/news/2020/03/24/demo-fraud-detection-2.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -369,6 +371,16 @@ This release marks a big milestone: Stateful Functions 2.0 is not only an API up
<ul id="markdown-toc">
+ <li><a href="/2021/01/18/rocksdb.html">Using RocksDB State Backend in Apache Flink: When and How</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/news/2021/01/11/batch-fine-grained-fault-tolerance.html">Exploring fine-grained recovery of bounded data sets on Flink</a></li>
diff --git a/content/blog/page5/index.html b/content/blog/page5/index.html
index fa1d200..0714180 100644
--- a/content/blog/page5/index.html
+++ b/content/blog/page5/index.html
@@ -201,6 +201,19 @@
<!-- Blog posts -->
<article>
+ <h2 class="blog-title"><a href="/news/2020/03/24/demo-fraud-detection-2.html">Advanced Flink Application Patterns Vol.2: Dynamic Updates of Application Logic</a></h2>
+
+ <p>24 Mar 2020
+ Alexander Fedulov (<a href="https://twitter.com/alex_fedulov">@alex_fedulov</a>)</p>
+
+ <p>In this series of blog posts you will learn about powerful Flink patterns for building streaming applications.</p>
+
+ <p><a href="/news/2020/03/24/demo-fraud-detection-2.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
<h2 class="blog-title"><a href="/ecosystem/2020/02/22/apache-beam-how-beam-runs-on-top-of-flink.html">Apache Beam: How Beam Runs on Top of Flink</a></h2>
<p>22 Feb 2020
@@ -323,19 +336,6 @@
<hr>
- <article>
- <h2 class="blog-title"><a href="/news/2019/11/25/query-pulsar-streams-using-apache-flink.html">How to query Pulsar Streams using Apache Flink</a></h2>
-
- <p>25 Nov 2019
- Sijie Guo (<a href="https://twitter.com/sijieg">@sijieg</a>) & Markos Sfikas (<a href="https://twitter.com/MarkSfik">@MarkSfik</a>)</p>
-
- <p>This blog post discusses the new developments and integrations between Apache Flink and Apache Pulsar and showcases how you can leverage Pulsar’s built-in schema to query Pulsar streams in real time using Apache Flink.</p>
-
- <p><a href="/news/2019/11/25/query-pulsar-streams-using-apache-flink.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -368,6 +368,16 @@
<ul id="markdown-toc">
+ <li><a href="/2021/01/18/rocksdb.html">Using RocksDB State Backend in Apache Flink: When and How</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/news/2021/01/11/batch-fine-grained-fault-tolerance.html">Exploring fine-grained recovery of bounded data sets on Flink</a></li>
diff --git a/content/blog/page6/index.html b/content/blog/page6/index.html
index 456e7f4..930fc7c 100644
--- a/content/blog/page6/index.html
+++ b/content/blog/page6/index.html
@@ -201,6 +201,19 @@
<!-- Blog posts -->
<article>
+ <h2 class="blog-title"><a href="/news/2019/11/25/query-pulsar-streams-using-apache-flink.html">How to query Pulsar Streams using Apache Flink</a></h2>
+
+ <p>25 Nov 2019
+ Sijie Guo (<a href="https://twitter.com/sijieg">@sijieg</a>) & Markos Sfikas (<a href="https://twitter.com/MarkSfik">@MarkSfik</a>)</p>
+
+ <p>This blog post discusses the new developments and integrations between Apache Flink and Apache Pulsar and showcases how you can leverage Pulsar’s built-in schema to query Pulsar streams in real time using Apache Flink.</p>
+
+ <p><a href="/news/2019/11/25/query-pulsar-streams-using-apache-flink.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
<h2 class="blog-title"><a href="/news/2019/10/18/release-1.9.1.html">Apache Flink 1.9.1 Released</a></h2>
<p>18 Oct 2019
@@ -326,19 +339,6 @@
<hr>
- <article>
- <h2 class="blog-title"><a href="/2019/05/19/state-ttl.html">State TTL in Flink 1.8.0: How to Automatically Cleanup Application State in Apache Flink</a></h2>
-
- <p>19 May 2019
- Fabian Hueske (<a href="https://twitter.com/fhueske">@fhueske</a>) & Andrey Zagrebin </p>
-
- <p>A common requirement for many stateful streaming applications is to automatically cleanup application state for effective management of your state size, or to control how long the application state can be accessed. State TTL enables application state cleanup and efficient state size management in Apache Flink</p>
-
- <p><a href="/2019/05/19/state-ttl.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -371,6 +371,16 @@
<ul id="markdown-toc">
+ <li><a href="/2021/01/18/rocksdb.html">Using RocksDB State Backend in Apache Flink: When and How</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/news/2021/01/11/batch-fine-grained-fault-tolerance.html">Exploring fine-grained recovery of bounded data sets on Flink</a></li>
diff --git a/content/blog/page7/index.html b/content/blog/page7/index.html
index 2babbef..c9bf9f7 100644
--- a/content/blog/page7/index.html
+++ b/content/blog/page7/index.html
@@ -201,6 +201,19 @@
<!-- Blog posts -->
<article>
+ <h2 class="blog-title"><a href="/2019/05/19/state-ttl.html">State TTL in Flink 1.8.0: How to Automatically Cleanup Application State in Apache Flink</a></h2>
+
+ <p>19 May 2019
+ Fabian Hueske (<a href="https://twitter.com/fhueske">@fhueske</a>) & Andrey Zagrebin </p>
+
+ <p>A common requirement for many stateful streaming applications is to automatically cleanup application state for effective management of your state size, or to control how long the application state can be accessed. State TTL enables application state cleanup and efficient state size management in Apache Flink</p>
+
+ <p><a href="/2019/05/19/state-ttl.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
<h2 class="blog-title"><a href="/2019/05/14/temporal-tables.html">Flux capacitor, huh? Temporal Tables and Joins in Streaming SQL</a></h2>
<p>14 May 2019
@@ -329,19 +342,6 @@ for more details.</p>
<hr>
- <article>
- <h2 class="blog-title"><a href="/news/2019/02/13/unified-batch-streaming-blink.html">Batch as a Special Case of Streaming and Alibaba's contribution of Blink</a></h2>
-
- <p>13 Feb 2019
- Stephan Ewen (<a href="https://twitter.com/stephanewen">@stephanewen</a>), Fabian Hueske (<a href="https://twitter.com/fhueske">@fhueske</a>), & Xiaowei Jiang (<a href="https://twitter.com/XiaoweiJ">@XiaoweiJ</a>)</p>
-
- <p>A few weeks ago, Alibaba contributed its Flink-fork 'Blink' back to Apache Flink. In this blog post we discuss how Blink's features will help the Flink community to make a big step towards its vision to build a truly unified system for stream and batch processing.</p>
-
- <p><a href="/news/2019/02/13/unified-batch-streaming-blink.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -374,6 +374,16 @@ for more details.</p>
<ul id="markdown-toc">
+ <li><a href="/2021/01/18/rocksdb.html">Using RocksDB State Backend in Apache Flink: When and How</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/news/2021/01/11/batch-fine-grained-fault-tolerance.html">Exploring fine-grained recovery of bounded data sets on Flink</a></li>
diff --git a/content/blog/page8/index.html b/content/blog/page8/index.html
index 34a7efe..592f7cf 100644
--- a/content/blog/page8/index.html
+++ b/content/blog/page8/index.html
@@ -201,6 +201,19 @@
<!-- Blog posts -->
<article>
+ <h2 class="blog-title"><a href="/news/2019/02/13/unified-batch-streaming-blink.html">Batch as a Special Case of Streaming and Alibaba's contribution of Blink</a></h2>
+
+ <p>13 Feb 2019
+ Stephan Ewen (<a href="https://twitter.com/stephanewen">@stephanewen</a>), Fabian Hueske (<a href="https://twitter.com/fhueske">@fhueske</a>), & Xiaowei Jiang (<a href="https://twitter.com/XiaoweiJ">@XiaoweiJ</a>)</p>
+
+ <p>A few weeks ago, Alibaba contributed its Flink-fork 'Blink' back to Apache Flink. In this blog post we discuss how Blink's features will help the Flink community to make a big step towards its vision to build a truly unified system for stream and batch processing.</p>
+
+ <p><a href="/news/2019/02/13/unified-batch-streaming-blink.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
<h2 class="blog-title"><a href="/news/2018/12/26/release-1.5.6.html">Apache Flink 1.5.6 Released</a></h2>
<p>26 Dec 2018
@@ -337,21 +350,6 @@ Please check the <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa
<hr>
- <article>
- <h2 class="blog-title"><a href="/news/2018/08/09/release-1.6.0.html">Apache Flink 1.6.0 Release Announcement</a></h2>
-
- <p>09 Aug 2018
- Till Rohrmann (<a href="https://twitter.com/stsffap">@stsffap</a>)</p>
-
- <p><p>The Apache Flink community is proud to announce the 1.6.0 release. Over the past 2 months, the Flink community has worked hard to resolve more than 360 issues. Please check the <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12342760">complete changelog</a> for more details.</p>
-
-</p>
-
- <p><a href="/news/2018/08/09/release-1.6.0.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -384,6 +382,16 @@ Please check the <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa
<ul id="markdown-toc">
+ <li><a href="/2021/01/18/rocksdb.html">Using RocksDB State Backend in Apache Flink: When and How</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/news/2021/01/11/batch-fine-grained-fault-tolerance.html">Exploring fine-grained recovery of bounded data sets on Flink</a></li>
diff --git a/content/blog/page9/index.html b/content/blog/page9/index.html
index f438966..8984fe1 100644
--- a/content/blog/page9/index.html
+++ b/content/blog/page9/index.html
@@ -201,6 +201,21 @@
<!-- Blog posts -->
<article>
+ <h2 class="blog-title"><a href="/news/2018/08/09/release-1.6.0.html">Apache Flink 1.6.0 Release Announcement</a></h2>
+
+ <p>09 Aug 2018
+ Till Rohrmann (<a href="https://twitter.com/stsffap">@stsffap</a>)</p>
+
+ <p><p>The Apache Flink community is proud to announce the 1.6.0 release. Over the past 2 months, the Flink community has worked hard to resolve more than 360 issues. Please check the <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12342760">complete changelog</a> for more details.</p>
+
+</p>
+
+ <p><a href="/news/2018/08/09/release-1.6.0.html">Continue reading »</a></p>
+ </article>
+
+ <hr>
+
+ <article>
<h2 class="blog-title"><a href="/news/2018/07/31/release-1.5.2.html">Apache Flink 1.5.2 Released</a></h2>
<p>31 Jul 2018
@@ -329,23 +344,6 @@
<hr>
- <article>
- <h2 class="blog-title"><a href="/news/2017/12/12/release-1.4.0.html">Apache Flink 1.4.0 Release Announcement</a></h2>
-
- <p>12 Dec 2017
- Aljoscha Krettek (<a href="https://twitter.com/aljoscha">@aljoscha</a>) & Mike Winters (<a href="https://twitter.com/wints">@wints</a>)</p>
-
- <p><p>The Apache Flink community is pleased to announce the 1.4.0 release. Over the past 5 months, the
-Flink community has been working hard to resolve more than 900 issues. See the <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12340533">complete changelog</a>
-for more detail.</p>
-
-</p>
-
- <p><a href="/news/2017/12/12/release-1.4.0.html">Continue reading »</a></p>
- </article>
-
- <hr>
-
<!-- Pagination links -->
@@ -378,6 +376,16 @@ for more detail.</p>
<ul id="markdown-toc">
+ <li><a href="/2021/01/18/rocksdb.html">Using RocksDB State Backend in Apache Flink: When and How</a></li>
+
+
+
+
+
+
+
+
+
<li><a href="/news/2021/01/11/batch-fine-grained-fault-tolerance.html">Exploring fine-grained recovery of bounded data sets on Flink</a></li>
diff --git a/content/img/blog/2021-01-18-rocksdb/RocksDB-in-Flink.png b/content/img/blog/2021-01-18-rocksdb/RocksDB-in-Flink.png
new file mode 100644
index 0000000..13115fb
Binary files /dev/null and b/content/img/blog/2021-01-18-rocksdb/RocksDB-in-Flink.png differ
diff --git a/content/index.html b/content/index.html
index f576e55..8cf396a 100644
--- a/content/index.html
+++ b/content/index.html
@@ -573,6 +573,9 @@
<dl>
+ <dt> <a href="/2021/01/18/rocksdb.html">Using RocksDB State Backend in Apache Flink: When and How</a></dt>
+ <dd>This blog post will guide you through the benefits of using RocksDB to manage your application’s state, explain when and how to use it and also clear up a few common misconceptions.</dd>
+
<dt> <a href="/news/2021/01/11/batch-fine-grained-fault-tolerance.html">Exploring fine-grained recovery of bounded data sets on Flink</a></dt>
<dd>Apache Flink 1.9 introduced fine-grained recovery through FLIP-1. The Flink APIs that are made for bounded workloads benefit from this change by individually recovering failed operators, re-using results from the previous processing step. This blog post gives an overview over these changes and evaluates their effectiveness.</dd>
@@ -588,9 +591,6 @@
<dd><p>The Apache Flink community released the third bugfix version of the Apache Flink 1.11 series.</p>
</dd>
-
- <dt> <a href="/2020/12/15/pipelined-region-sheduling.html">Improvements in task scheduling for batch workloads in Apache Flink 1.12</a></dt>
- <dd>In this blogpost, we’ll take a closer look at how far the community has come in improving task scheduling for batch workloads, why this matters and what you can expect in Flink 1.12 with the new pipelined region scheduler.</dd>
</dl>
diff --git a/content/zh/index.html b/content/zh/index.html
index 0e88971..43d2c32 100644
--- a/content/zh/index.html
+++ b/content/zh/index.html
@@ -570,6 +570,9 @@
<dl>
+ <dt> <a href="/2021/01/18/rocksdb.html">Using RocksDB State Backend in Apache Flink: When and How</a></dt>
+ <dd>This blog post will guide you through the benefits of using RocksDB to manage your application’s state, explain when and how to use it and also clear up a few common misconceptions.</dd>
+
<dt> <a href="/news/2021/01/11/batch-fine-grained-fault-tolerance.html">Exploring fine-grained recovery of bounded data sets on Flink</a></dt>
<dd>Apache Flink 1.9 introduced fine-grained recovery through FLIP-1. The Flink APIs that are made for bounded workloads benefit from this change by individually recovering failed operators, re-using results from the previous processing step. This blog post gives an overview over these changes and evaluates their effectiveness.</dd>
@@ -585,9 +588,6 @@
<dd><p>The Apache Flink community released the third bugfix version of the Apache Flink 1.11 series.</p>
</dd>
-
- <dt> <a href="/2020/12/15/pipelined-region-sheduling.html">Improvements in task scheduling for batch workloads in Apache Flink 1.12</a></dt>
- <dd>In this blogpost, we’ll take a closer look at how far the community has come in improving task scheduling for batch workloads, why this matters and what you can expect in Flink 1.12 with the new pipelined region scheduler.</dd>
</dl>
[flink-web] 03/10: cloud based -> cloud-based
Posted by rm...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
rmetzger pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/flink-web.git
commit 635c74a09130e90263ad9a7e5110c1906cd9fc33
Author: qinjunjerry <11...@users.noreply.github.com>
AuthorDate: Tue Jan 12 13:55:55 2021 +0100
cloud based -> cloud-based
Co-authored-by: morsapaes <ma...@gmail.com>
---
_posts/2020-12-20-rocksdb.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/_posts/2020-12-20-rocksdb.md b/_posts/2020-12-20-rocksdb.md
index a12016a..2d74e14 100644
--- a/_posts/2020-12-20-rocksdb.md
+++ b/_posts/2020-12-20-rocksdb.md
@@ -81,7 +81,7 @@ Add the following into your job’s code after StreamExecutionEnvironment is cre
<div class="alert alert-info" markdown="1">
<span class="label label-info" style="display: inline-block"><span class="glyphicon glyphicon-info-sign" aria-hidden="true"></span> Note</span>
-In addition to HDFS, you can also use other on-premises or cloud based object stores if the corresponding dependencies are added under [FLINK_HOME/plugins](https://ci.apache.org/projects/flink/flink-docs-stable/deployment/filesystems/plugins.html).
+In addition to HDFS, you can also use other on-premises or cloud-based object stores if the corresponding dependencies are added under [FLINK_HOME/plugins](https://ci.apache.org/projects/flink/flink-docs-stable/deployment/filesystems/plugins.html).
</div>
# Best Practices and Advanced Configuration
[flink-web] 09/10: rename file/dir to 2021-01-18
Posted by rm...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
rmetzger pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/flink-web.git
commit e93a79db4ae89242e121772bdbc21c4306dc29f1
Author: Jun Qin <11...@users.noreply.github.com>
AuthorDate: Tue Jan 12 14:57:03 2021 +0100
rename file/dir to 2021-01-18
---
_posts/{2020-12-20-rocksdb.md => 2021-01-18-rocksdb.md} | 0
.../RocksDB-in-Flink.png | Bin
2 files changed, 0 insertions(+), 0 deletions(-)
diff --git a/_posts/2020-12-20-rocksdb.md b/_posts/2021-01-18-rocksdb.md
similarity index 100%
rename from _posts/2020-12-20-rocksdb.md
rename to _posts/2021-01-18-rocksdb.md
diff --git a/img/blog/2020-12-20-rocksdb/RocksDB-in-Flink.png b/img/blog/2021-01-18-rocksdb/RocksDB-in-Flink.png
similarity index 100%
rename from img/blog/2020-12-20-rocksdb/RocksDB-in-Flink.png
rename to img/blog/2021-01-18-rocksdb/RocksDB-in-Flink.png
[flink-web] 06/10: to enable -> enabling
Posted by rm...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
rmetzger pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/flink-web.git
commit 1bf7987b444d6dc5d3bbd8a2351ace4f4b9c01e6
Author: qinjunjerry <11...@users.noreply.github.com>
AuthorDate: Tue Jan 12 14:22:42 2021 +0100
to enable -> enabling
Co-authored-by: morsapaes <ma...@gmail.com>
---
_posts/2020-12-20-rocksdb.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/_posts/2020-12-20-rocksdb.md b/_posts/2020-12-20-rocksdb.md
index 84f8d31..17e26d6 100644
--- a/_posts/2020-12-20-rocksdb.md
+++ b/_posts/2020-12-20-rocksdb.md
@@ -112,7 +112,7 @@ Enabling RocksDB's native metrics in Flink may have a negative performance impac
Since Flink 1.10, Flink configures RocksDB’s memory allocation to the amount of managed memory of each task slot by default. The primary mechanism for improving memory-related performance issues is to increase Flink’s [managed memory](https://ci.apache.org/projects/flink/flink-docs-stable/deployment/memory/mem_setup_tm.html#managed-memory) via the Flink configuration [`taskmanager.memory.managed.size`](https://ci.apache.org/projects/flink/flink-docs-stable/deployment/config.html#taskmana [...]
-While data is being written or overwritten in RocksDB, flushing from memory to local disks and data compaction are managed in the background by RocksDB threads. On a machine with many CPU cores, you should increase the parallelism of background flushing and compaction by setting the Flink configuration [`state.backend.rocksdb.thread.num`](https://ci.apache.org/projects/flink/flink-docs-stable/deployment/config.html#state-backend-rocksdb-thread-num) (corresponding to max_background_jobs i [...]
+While data is being written or overwritten in RocksDB, flushing from memory to local disks and data compaction are managed in the background by RocksDB threads. On a machine with many CPU cores, you should increase the parallelism of background flushing and compaction by setting the Flink configuration [`state.backend.rocksdb.thread.num`](https://ci.apache.org/projects/flink/flink-docs-stable/deployment/config.html#state-backend-rocksdb-thread-num) (corresponding to max_background_jobs i [...]
For other RocksDBStateBackend configurations, check the Flink documentation on [Advanced RocksDB State Backends Options](https://ci.apache.org/projects/flink/flink-docs-stable/deployment/config.html#advanced-rocksdb-state-backends-options). For further tuning, check [RocksDB Tuning Guide](https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide) in [RocksDB Wiki](https://github.com/facebook/rocksdb/wiki).
[flink-web] 05/10: Cover also Flink 1.12
Posted by rm...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
rmetzger pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/flink-web.git
commit ab8d20d63109a42bee56671c015291c785f29680
Author: qinjunjerry <11...@users.noreply.github.com>
AuthorDate: Tue Jan 12 14:17:14 2021 +0100
Cover also Flink 1.12
Co-authored-by: morsapaes <ma...@gmail.com>
---
_posts/2020-12-20-rocksdb.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/_posts/2020-12-20-rocksdb.md b/_posts/2020-12-20-rocksdb.md
index 2252086..84f8d31 100644
--- a/_posts/2020-12-20-rocksdb.md
+++ b/_posts/2020-12-20-rocksdb.md
@@ -100,7 +100,7 @@ To check how RocksDB is behaving in production, you should look for the RocksDB
<div class="alert alert-info" markdown="1">
<span class="label label-info" style="display: inline-block"><span class="glyphicon glyphicon-info-sign" aria-hidden="true"></span> Note</span>
-Flink 1.10 & 1.11 effectively disabled RocksDB logging by [setting the log level to HEADER](https://github.com/apache/flink/blob/master/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/PredefinedOptions.java#L64). To enable it, check [How to get RocksDB's LOG file back for advanced troubleshooting](https://ververica.zendesk.com/hc/en-us/articles/360015933320-How-to-get-RocksDB-s-LOG-file-back-for-advanced-troubleshooting).
+From Flink 1.10, RocksDB logging was effectively disabled by [setting the log level to HEADER](https://github.com/apache/flink/blob/master/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/PredefinedOptions.java#L64). To enable it, check [How to get RocksDB's LOG file back for advanced troubleshooting](https://ververica.zendesk.com/hc/en-us/articles/360015933320-How-to-get-RocksDB-s-LOG-file-back-for-advanced-troubleshooting).
</div>
<div class="alert alert-warning" markdown="1">
[flink-web] 02/10: spill to disks -> spill to disk
Posted by rm...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
rmetzger pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/flink-web.git
commit 5e0487e0bf8d16dfb1a570b707f45c46bfb129eb
Author: qinjunjerry <11...@users.noreply.github.com>
AuthorDate: Tue Jan 12 13:44:23 2021 +0100
spill to disks -> spill to disk
Co-authored-by: morsapaes <ma...@gmail.com>
---
_posts/2020-12-20-rocksdb.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/_posts/2020-12-20-rocksdb.md b/_posts/2020-12-20-rocksdb.md
index e2bb51b..a12016a 100644
--- a/_posts/2020-12-20-rocksdb.md
+++ b/_posts/2020-12-20-rocksdb.md
@@ -12,7 +12,7 @@ excerpt: This blog post will guide you through the benefits of using RocksDB to
# State in Flink
-To best understand state and state backends in Flink, it’s important to distinguish between **in-flight state** and **state snapshots**. In-flight state, also known as working state, is the state a Flink job is working on. It is always stored locally in memory (with the possibility to spill to disks) and can be lost when jobs fail without impacting job recoverability. State snapshots, i.e., [checkpoints](https://ci.apache.org/projects/flink/flink-docs-stable/ops/state/checkpoints.html) a [...]
+To best understand state and state backends in Flink, it’s important to distinguish between **in-flight state** and **state snapshots**. In-flight state, also known as working state, is the state a Flink job is working on. It is always stored locally in memory (with the possibility to spill to disk) and can be lost when jobs fail without impacting job recoverability. State snapshots, i.e., [checkpoints](https://ci.apache.org/projects/flink/flink-docs-stable/ops/state/checkpoints.html) an [...]
# What is RocksDB?
[flink-web] 01/10: Add the rocksdb blog
Posted by rm...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
rmetzger pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/flink-web.git
commit 50d21735bdc3669384385fa511f8aa145618b0f1
Author: Jun Qin <11...@users.noreply.github.com>
AuthorDate: Sat Dec 19 23:51:32 2020 +0100
Add the rocksdb blog
---
_posts/2020-12-20-rocksdb.md | 121 +++++++++++++++++++++++
img/blog/2020-12-20-rocksdb/RocksDB-in-Flink.png | Bin 0 -> 72641 bytes
2 files changed, 121 insertions(+)
diff --git a/_posts/2020-12-20-rocksdb.md b/_posts/2020-12-20-rocksdb.md
new file mode 100644
index 0000000..e2bb51b
--- /dev/null
+++ b/_posts/2020-12-20-rocksdb.md
@@ -0,0 +1,121 @@
+---
+layout: post
+title: "Using RocksDB State Backend in Apache Flink: When and How"
+date: 2020-12-20 00:00:00
+authors:
+- Jun Qin:
+ name: "Jun Qin"
+excerpt: This blog post will guide you through the benefits of using RocksDB to manage your application’s state, explain when and how to use it and also clear up a few common misconceptions.
+---
+
+[Stream processing](https://en.wikipedia.org/wiki/Stream_processing) applications are often stateful, “remembering” information from processed events and using it to influence further event processing. In Flink, the remembered information, i.e., state, is stored locally in the configured state backend. To prevent data loss in case of failures, the state backend periodically persists a snapshot of its contents to a pre-configured durable storage. The [RocksDB](https://rocksdb.org/) state [...]
+
+# State in Flink
+
+To best understand state and state backends in Flink, it’s important to distinguish between **in-flight state** and **state snapshots**. In-flight state, also known as working state, is the state a Flink job is working on. It is always stored locally in memory (with the possibility to spill to disks) and can be lost when jobs fail without impacting job recoverability. State snapshots, i.e., [checkpoints](https://ci.apache.org/projects/flink/flink-docs-stable/ops/state/checkpoints.html) a [...]
+
+# What is RocksDB?
+
+Thinking of RocksDB as a distributed database that needs to run on a cluster and to be managed by specialized administrators is a common misconception. RocksDB is an embeddable persistent key-value store for fast storage. It interacts with Flink via the Java Native Interface (JNI). The picture below shows where RocksDB fits in a Flink cluster node. Details are explained in the following sections.
+
+
+<center>
+<img vspace="8" style="width:60%" src="{{site.baseurl}}/img/blog/2020-12-20-rocksdb/RocksDB-in-Flink.png" />
+</center>
+
+
+# RocksDB in Flink
+
+Everything you need to use RocksDB as a state backend is bundled in the Apache Flink distribution, including the native shared library:
+
+ $ jar -tvf lib/flink-dist_2.12-1.12.0.jar| grep librocksdbjni-linux64
+ 8695334 Wed Nov 27 02:27:06 CET 2019 librocksdbjni-linux64.so
+
+At runtime, RocksDB is embedded in the TaskManager processes. It runs in native threads and works with local files. For example, if you have a job configured with RocksDBStateBackend running in your Flink cluster, you’ll see something similar to the following, where 32513 is the TaskManager process ID.
+
+ $ ps -T -p 32513 | grep -i rocksdb
+ 32513 32633 ? 00:00:00 rocksdb:low0
+ 32513 32634 ? 00:00:00 rocksdb:high0
+
+<div class="alert alert-info" markdown="1">
+<span class="label label-info" style="display: inline-block"><span class="glyphicon glyphicon-info-sign" aria-hidden="true"></span> Note</span>
+The command is for Linux only. For other operating systems, please refer to their documentation.
+</div>
+
+
+# When to use RocksDBStateBackend
+
+In addition to RocksDBStateBackend, Flink has two other built-in state backends: MemoryStateBackend and FsStateBackend. They both are heap-based, as in-flight state is stored in the JVM heap. For the moment being, let’s ignore MemoryStateBackend, as it is intended only for **local developments** and **debugging**, not for production use.
+
+With RocksDBStateBackend, in-flight state is first written into off-heap/native memory, and then flushed to local disks when a configured threshold is reached. This means that RocksDBStateBackend can support state larger than the total configured heap capacity. The amount of state that you can store in RocksDBStateBackend is only limited by the amount of **disk space** available across the entire cluster. In addition, since RocksDBStateBackend doesn’t use JVM heap to store in-flight sta [...]
+
+On top of full, self-contained state snapshots, RocksDBStateBackend also supports [incremental checkpointing](https://ci.apache.org/projects/flink/flink-docs-stable/ops/state/large_state_tuning.html#incremental-checkpoints) as a performance tuning option. An incremental checkpoint stores only the changes that occurred since the latest completed checkpoint. This dramatically reduces checkpointing time in comparison to performing a full snapshot. RocksDBStateBackend is currently the only s [...]
+
+RocksDB is a good option when:
+
+- The state of your job is larger than can fit in local memory (e.g., long windows, large [keyed state](https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/state.html));
+- You’re looking into incremental checkpointing as a way to reduce checkpointing time;
+- You expect to have more predictable latency without being impacted by JVM Garbage Collection.
+
+Otherwise, if your application has **small state** or requires **very low latency**, you should consider **FsStateBackend**. As a rule of thumb, RocksDBStateBackend is a few times slower than heap-based state backends, because it stores key/value pairs as serialized bytes. This means that any state access (reads/writes) needs to go through a de-/serialization process crossing the JNI boundary, which is more expensive than working directly with the on-heap representation of state. The ups [...]
+
+# How to use RocksDBStateBackend
+
+RocksDB is fully embedded within and fully managed by the TaskManager process. RocksDBStateBackend can be configured at the cluster level as the default for the entire cluster, or at the job level for individual jobs. The job level configuration takes precedence over the cluster level configuration.
+
+## Cluster Level
+
+Add the following configuration in [`conf/flink-conf.yaml`](https://ci.apache.org/projects/flink/flink-docs-stable/deployment/config.html):
+
+ state.backend: rocksdb
+ state.backend.incremental: true
+ state.checkpoints.dir: hdfs:///flink-checkpoints # location to store checkpoints
+
+# Job Level
+
+Add the following into your job’s code after StreamExecutionEnvironment is created:
+
+ # 'env' is the created StreamExecutionEnvironment
+ # 'true' is to enable incremental checkpointing
+ env.setStateBackend(new RocksDBStateBackend("hdfs:///fink-checkpoints", true));
+
+<div class="alert alert-info" markdown="1">
+<span class="label label-info" style="display: inline-block"><span class="glyphicon glyphicon-info-sign" aria-hidden="true"></span> Note</span>
+In addition to HDFS, you can also use other on-premises or cloud based object stores if the corresponding dependencies are added under [FLINK_HOME/plugins](https://ci.apache.org/projects/flink/flink-docs-stable/deployment/filesystems/plugins.html).
+</div>
+
+# Best Practices and Advanced Configuration
+
+We hope this overview helped you gain a better understanding of the role of RocksDB in Flink and how to successfully run a job with RocksDBStateBackend. To round it off, we’ll explore some best practices and a few reference points for further troubleshooting and performance tuning.
+
+## State Location in RocksDB
+
+As mentioned earlier, in-flight state in RocksDBStateBackend is spilled to files on disks. These files are located under the directory specified by the Flink configuration [`state.backend.rocksdb.localdir`](https://ci.apache.org/projects/flink/flink-docs-stable/deployment/config.html#state-backend-rocksdb-localdir). Because disk performance has a direct impact on RocksDB’s performance, it’s recommended that this directory is located on a **local** disk. It’s discouraged to configure it [...]
+
+State snapshots are persisted to remote durable storage. During state snapshotting, TaskManagers take a snapshot of the in-flight state and store it remotely. Transferring the state snapshot to remote storage is handled purely by the TaskManager itself without the involvement of the state backend. So, [`state.checkpoints.dir`](https://ci.apache.org/projects/flink/flink-docs-stable/deployment/config.html#state-checkpoints-dir) or the parameter you set in the code for a particular job can [...]
+
+## Troubleshooting RocksDB
+
+To check how RocksDB is behaving in production, you should look for the RocksDB log file named LOG. By default, this log file is located in the same directory as your data files, i.e., the directory specified by the Flink configuration [`state.backend.rocksdb.localdir`](https://ci.apache.org/projects/flink/flink-docs-stable/deployment/config.html#state-backend-rocksdb-localdir). When enabled, [RocksDB statistics](https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide#rocksdb-stati [...]
+
+<div class="alert alert-info" markdown="1">
+<span class="label label-info" style="display: inline-block"><span class="glyphicon glyphicon-info-sign" aria-hidden="true"></span> Note</span>
+Flink 1.10 & 1.11 effectively disabled RocksDB logging by [setting the log level to HEADER](https://github.com/apache/flink/blob/master/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/PredefinedOptions.java#L64). To enable it, check [How to get RocksDB's LOG file back for advanced troubleshooting](https://ververica.zendesk.com/hc/en-us/articles/360015933320-How-to-get-RocksDB-s-LOG-file-back-for-advanced-troubleshooting).
+</div>
+
+<div class="alert alert-warning" markdown="1">
+<span class="label label-warning" style="display: inline-block"><span class="glyphicon glyphicon-warning-sign" aria-hidden="true"></span> Warning</span>
+Enabling RocksDB's native metrics in Flink may have a negative performance impact on your job.
+</div>
+
+## Tuning RocksDB
+
+Since Flink 1.10, Flink configures RocksDB’s memory allocation to the amount of managed memory of each task slot by default. The primary mechanism for improving memory-related performance issues is to increase Flink’s [managed memory](https://ci.apache.org/projects/flink/flink-docs-stable/deployment/memory/mem_setup_tm.html#managed-memory) via the Flink configuration [`taskmanager.memory.managed.size`](https://ci.apache.org/projects/flink/flink-docs-stable/deployment/config.html#taskmana [...]
+
+While data is being written or overwritten in RocksDB, flushing from memory to local disks and data compaction are managed in the background by RocksDB threads. On a machine with many CPU cores, you should increase the parallelism of background flushing and compaction by setting the Flink configuration [`state.backend.rocksdb.thread.num`](https://ci.apache.org/projects/flink/flink-docs-stable/deployment/config.html#state-backend-rocksdb-thread-num) (corresponding to max_background_jobs i [...]
+
+For other RocksDBStateBackend configurations, check the Flink documentation on [Advanced RocksDB State Backends Options](https://ci.apache.org/projects/flink/flink-docs-stable/deployment/config.html#advanced-rocksdb-state-backends-options). For further tuning, check [RocksDB Tuning Guide](https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide) in [RocksDB Wiki](https://github.com/facebook/rocksdb/wiki).
+
+# Conclusion
+
+The [RocksDB](https://rocksdb.org/) state backend (i.e., RocksDBStateBackend) is one of the three state backends bundled in Flink, and can be a powerful choice when configuring your streaming applications. It enables scalable applications maintaining up to many terabytes of state with exactly-once processing guarantees. If the state of your Flink job is too large to fit on the JVM heap, you are interested in incremental checkpointing, or you expect to have predictable latency, you shoul [...]
diff --git a/img/blog/2020-12-20-rocksdb/RocksDB-in-Flink.png b/img/blog/2020-12-20-rocksdb/RocksDB-in-Flink.png
new file mode 100644
index 0000000..13115fb
Binary files /dev/null and b/img/blog/2020-12-20-rocksdb/RocksDB-in-Flink.png differ
[flink-web] 08/10: change date to 2021-01-18
Posted by rm...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
rmetzger pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/flink-web.git
commit 185bb1a97463424be9be8875d0a7f10de5a776d2
Author: Jun Qin <11...@users.noreply.github.com>
AuthorDate: Tue Jan 12 14:54:20 2021 +0100
change date to 2021-01-18
---
_posts/2020-12-20-rocksdb.md | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/_posts/2020-12-20-rocksdb.md b/_posts/2020-12-20-rocksdb.md
index 071a9c0..472cba6 100644
--- a/_posts/2020-12-20-rocksdb.md
+++ b/_posts/2020-12-20-rocksdb.md
@@ -1,14 +1,14 @@
---
layout: post
title: "Using RocksDB State Backend in Apache Flink: When and How"
-date: 2020-12-20 00:00:00
+date: 2021-01-18 00:00:00
authors:
- Jun Qin:
name: "Jun Qin"
excerpt: This blog post will guide you through the benefits of using RocksDB to manage your application’s state, explain when and how to use it and also clear up a few common misconceptions.
---
-[Stream processing](https://en.wikipedia.org/wiki/Stream_processing) applications are often stateful, “remembering” information from processed events and using it to influence further event processing. In Flink, the remembered information, i.e., state, is stored locally in the configured state backend. To prevent data loss in case of failures, the state backend periodically persists a snapshot of its contents to a pre-configured durable storage. The [RocksDB](https://rocksdb.org/) state [...]
+Stream processing applications are often stateful, “remembering” information from processed events and using it to influence further event processing. In Flink, the remembered information, i.e., state, is stored locally in the configured state backend. To prevent data loss in case of failures, the state backend periodically persists a snapshot of its contents to a pre-configured durable storage. The [RocksDB](https://rocksdb.org/) state backend (i.e., RocksDBStateBackend) is one of the t [...]
# State in Flink
@@ -20,7 +20,7 @@ Thinking of RocksDB as a distributed database that needs to run on a cluster and
<center>
-<img vspace="8" style="width:60%" src="{{site.baseurl}}/img/blog/2020-12-20-rocksdb/RocksDB-in-Flink.png" />
+<img vspace="8" style="width:60%" src="{{site.baseurl}}/img/blog/2021-01-18-rocksdb/RocksDB-in-Flink.png" />
</center>