You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by fa...@apache.org on 2023/11/26 02:07:28 UTC
(flink) branch master updated (27ada9bee73 -> 09702a22e43)
This is an automated email from the ASF dual-hosted git repository.
fanrui pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git
from 27ada9bee73 [hotfix][doc] remove hive connector repo link because it is still WIP
new e4f9cce7ed2 [FLINK-33650][doc] Remove manage memory part in FileSystemCheckpointStorage due to it can be used for rocksdb state backend.
new 09702a22e43 [FLINK-33650][doc] Add the Checkpoint Storage part for Chinese Checkpoint page
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
docs/content.zh/docs/ops/state/checkpoints.md | 52 +++++++++++++++++++++++++++
docs/content/docs/ops/state/checkpoints.md | 2 --
2 files changed, 52 insertions(+), 2 deletions(-)
(flink) 02/02: [FLINK-33650][doc] Add the Checkpoint Storage part for Chinese Checkpoint page
Posted by fa...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
fanrui pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git
commit 09702a22e43fddfa1ac78e2b8db92c2d1f5395a6
Author: 1996fanrui <19...@gmail.com>
AuthorDate: Sat Nov 25 20:39:18 2023 +0800
[FLINK-33650][doc] Add the Checkpoint Storage part for Chinese Checkpoint page
---
docs/content.zh/docs/ops/state/checkpoints.md | 52 +++++++++++++++++++++++++++
1 file changed, 52 insertions(+)
diff --git a/docs/content.zh/docs/ops/state/checkpoints.md b/docs/content.zh/docs/ops/state/checkpoints.md
index 732c9bf5a1e..616a272953f 100644
--- a/docs/content.zh/docs/ops/state/checkpoints.md
+++ b/docs/content.zh/docs/ops/state/checkpoints.md
@@ -33,6 +33,58 @@ Checkpoint 使 Flink 的状态具有良好的容错性，通过 checkpoint 机
要了解 checkpoints 和 [savepoints]({{< ref "docs/ops/state/savepoints" >}}) 之间的区别，请参阅 [checkpoints 与 savepoints]({{< ref "docs/ops/state/checkpoints_vs_savepoints" >}})。
+## Checkpoint 存储
+
+启用 Checkpoint 后，managed State 将被持久化，来确保故障后 Flink job 恢复的一致性。
+Checkpoint 期间 State 存储的位置取决于所选的 **Checkpoint 存储**。
+
+## 可选的 Checkpoint 存储
+
+Flink 开箱即用地提供了两种 Checkpoint 存储类型：
+
+- *JobManagerCheckpointStorage*
+- *FileSystemCheckpointStorage*
+
+{{< hint info >}}
+如果配置了 Checkpoint 目录，将使用 `FileSystemCheckpointStorage`，否则系统将使用 `JobManagerCheckpointStorage`。
+{{< /hint >}}
+
+### JobManagerCheckpointStorage
+
+*JobManagerCheckpointStorage* 将 Checkpoint 快照存储在 JobManager 的堆内存中。
+
+可以将其配置为在超过一定大小时使 Checkpoint 失败，以避免 JobManager 出现 `OutOfMemoryError`。
+要设置此功能，用户可以实例化具有相应最大大小的 `JobManagerCheckpointStorage`：
+
+```java
+new JobManagerCheckpointStorage(MAX_MEM_STATE_SIZE);
+```
+
+`JobManagerCheckpointStorage` 的限制:
+
+- 默认情况下，每个 State 的大小限制为 5 MB。 可以在 `JobManagerCheckpointStorage` 的构造函数中修改大小。
+- 无论配置的最大 State 大小如何，状态都不能大于 Pekka 框架的大小（请参阅 [配置参数]({{< ref "docs/deployment/config" >}})）。
+- 聚合后总的状态大小必须小于 JobManager 的内存上限。
+
+鼓励在以下场景使用 JobManagerCheckpointStorage：
+
+- 本地开发和调试
+- 使用很少状态的作业，例如仅包含每次仅存储一条记录（Map、FlatMap、Filter...）的作业。 Kafka 消费者需要很少的 State。
+
+### FileSystemCheckpointStorage
+
+*FileSystemCheckpointStorage* 配置中包含文件系统 URL（类型、地址、路径），
+例如 "hdfs://namenode:40010/flink/checkpoints" 或 "file:///data/flink/checkpoints"。
+
+Checkpoint 时， Flink 会将 State 快照写到配置的文件系统和目录的文件中。
+最少的元数据存储在 JobManager 的内存中（或者，高可用性模式下存储在 Checkpoint 的元数据中）。
+
+如果指定了 Checkpoint 目录，`FileSystemCheckpointStorage` 将用于保存 Checkpoint 快照。
+
+鼓励使用 `FileSystemCheckpointStorage` 的场景：
+
+- 所有高可用的场景。
+
## 保留 Checkpoint
Checkpoint 在默认的情况下仅用于恢复失败的作业，并不保留，当程序取消时 checkpoint 就会被删除。当然，你可以通过配置来保留 checkpoint，这些被保留的 checkpoint 在作业失败或取消时不会被清除。这样，你就可以使用该 checkpoint 来恢复失败的作业。
(flink) 01/02: [FLINK-33650][doc] Remove manage memory part in FileSystemCheckpointStorage due to it can be used for rocksdb state backend.
Posted by fa...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
fanrui pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git
commit e4f9cce7ed20c736ca1f52f2947a45301ef2a069
Author: 1996fanrui <19...@gmail.com>
AuthorDate: Sat Nov 25 20:34:34 2023 +0800
[FLINK-33650][doc] Remove manage memory part in FileSystemCheckpointStorage due to it can be used for rocksdb state backend.
---
docs/content/docs/ops/state/checkpoints.md | 2 --
1 file changed, 2 deletions(-)
diff --git a/docs/content/docs/ops/state/checkpoints.md b/docs/content/docs/ops/state/checkpoints.md
index 961db367fc4..12e9416c5c2 100644
--- a/docs/content/docs/ops/state/checkpoints.md
+++ b/docs/content/docs/ops/state/checkpoints.md
@@ -86,8 +86,6 @@ The `FileSystemCheckpointStorage` is encouraged for:
- All high-availability setups.
-It is also recommended to set [managed memory]({{< ref "docs/deployment/memory/mem_setup_tm" >}}#managed-memory) to zero.
-This will ensure that the maximum amount of memory is allocated for user code on the JVM.
## Retained Checkpoints