You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by pn...@apache.org on 2022/03/30 12:10:09 UTC
[flink] branch release-1.15 updated: [FLINK-26134][docs] Added documentation page with Checkpoint/Savepoint guarantees
This is an automated email from the ASF dual-hosted git repository.
pnowojski pushed a commit to branch release-1.15
in repository https://gitbox.apache.org/repos/asf/flink.git
The following commit(s) were added to refs/heads/release-1.15 by this push:
new b78aff5 [FLINK-26134][docs] Added documentation page with Checkpoint/Savepoint guarantees
b78aff5 is described below
commit b78aff52b00a43e6f9975a9b55847e1573f9f241
Author: Anton Kalashnikov <ka...@yandex.ru>
AuthorDate: Mon Feb 14 17:26:58 2022 +0100
[FLINK-26134][docs] Added documentation page with Checkpoint/Savepoint guarantees
---
.../docs/concepts/stateful-stream-processing.md | 5 +-
docs/content.zh/docs/ops/state/checkpoints.md | 8 +-
.../docs/ops/state/checkpoints_vs_savepoints.md | 95 ++++++++++++++++++++++
docs/content.zh/docs/ops/state/savepoints.md | 8 +-
.../docs/concepts/stateful-stream-processing.md | 3 +
docs/content/docs/ops/state/checkpoints.md | 10 +--
.../docs/ops/state/checkpoints_vs_savepoints.md | 95 ++++++++++++++++++++++
docs/content/docs/ops/state/savepoints.md | 11 +--
8 files changed, 205 insertions(+), 30 deletions(-)
diff --git a/docs/content.zh/docs/concepts/stateful-stream-processing.md b/docs/content.zh/docs/concepts/stateful-stream-processing.md
index 011c96b..d4748b5 100644
--- a/docs/content.zh/docs/concepts/stateful-stream-processing.md
+++ b/docs/content.zh/docs/concepts/stateful-stream-processing.md
@@ -310,7 +310,10 @@ mechanism for this.
Savepoints are similar to checkpoints except that they are
**triggered by the user** and **don't automatically expire** when newer
-checkpoints are completed.
+checkpoints are completed.
+To make proper use of savepoints, it's important to understand the differences between
+[checkpoints]({{< ref "docs/ops/state/checkpoints" >}}) and [savepoints]({{< ref "docs/ops/state/savepoints" >}})
+which is described in [checkpoints vs. savepoints]({{< ref "docs/ops/state/checkpoints_vs_savepoints" >}}).
{{< top >}}
diff --git a/docs/content.zh/docs/ops/state/checkpoints.md b/docs/content.zh/docs/ops/state/checkpoints.md
index 9fb8e64..f720388 100644
--- a/docs/content.zh/docs/ops/state/checkpoints.md
+++ b/docs/content.zh/docs/ops/state/checkpoints.md
@@ -31,6 +31,8 @@ Checkpoint 使 Flink 的状态具有良好的容错性，通过 checkpoint 机
参考 [Checkpointing]({{< ref "docs/dev/datastream/fault-tolerance/checkpointing" >}}) 查看如何在 Flink 程序中开启和配置 checkpoint。
+To understand the differences between checkpoints and [savepoints]({{< ref "docs/ops/state/savepoints" >}}) see [checkpoints vs. savepoints]({{< ref "docs/ops/state/checkpoints_vs_savepoints" >}}).
+
## 保留 Checkpoint
Checkpoint 在默认的情况下仅用于恢复失败的作业，并不保留，当程序取消时 checkpoint 就会被删除。当然，你可以通过配置来保留 checkpoint，这些被保留的 checkpoint 在作业失败或取消时不会被清除。这样，你就可以使用该 checkpoint 来恢复失败的作业。
@@ -80,12 +82,6 @@ state.checkpoints.dir: hdfs:///checkpoints/
env.setStateBackend(new RocksDBStateBackend("hdfs:///checkpoints-data/"));
```
-### Checkpoint 与 Savepoint 的区别
-
-Checkpoint 与 [savepoints]({{< ref "docs/ops/state/savepoints" >}}) 有一些区别，体现在 checkpoint ：
-- 使用 state backend 特定的数据格式，可能以增量方式存储。
-- 不支持 Flink 的特定功能，比如扩缩容。
-
### 从保留的 checkpoint 中恢复状态
与 savepoint 一样，作业可以从 checkpoint 的元数据文件恢复运行（[savepoint恢复指南]({{< ref "docs/ops/state/savepoints" >}}#resuming-from-savepoints)）。注意，如果元数据文件中信息不充分，那么 jobmanager 就需要使用相关的数据文件来恢复作业(参考[目录结构](#directory-structure))。
diff --git a/docs/content.zh/docs/ops/state/checkpoints_vs_savepoints.md b/docs/content.zh/docs/ops/state/checkpoints_vs_savepoints.md
new file mode 100644
index 0000000..0b66b25
--- /dev/null
+++ b/docs/content.zh/docs/ops/state/checkpoints_vs_savepoints.md
@@ -0,0 +1,95 @@
+---
+title: "Checkpoints vs. Savepoints"
+weight: 10
+type: docs
+aliases:
+ - /ops/state/checkpoints_vs_savepoints.html
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Checkpoints vs. Savepoints
+
+## Overview
+
+Conceptually, Flink's [savepoints]({{< ref "docs/ops/state/savepoints" >}}) are different from [checkpoints]({{< ref "docs/ops/state/checkpoints" >}})
+in a way that's analogous to how backups are different from recovery logs in traditional database systems.
+
+The primary purpose of checkpoints is to provide a recovery mechanism in case of unexpected job failures.
+A [checkpoint's lifecycle]({{< ref "docs/dev/datastream/fault-tolerance/checkpointing" >}}) is managed by Flink,
+i.e. a checkpoint is created, owned, and released by Flink - without user interaction.
+Because checkpoints are being triggered often, and are relied upon for failure recovery, the two main design goals for the checkpoint implementation are
+i) being as lightweight to create and ii) being as fast to restore from as possible.
+Optimizations towards those goals can exploit certain properties, e.g., that the job code doesn't change between the execution attempts.
+
+{{< hint info >}}
+- Checkpoints are automatically deleted if the application is terminated by the user
+(except if checkpoints are explicitly configured to be retained).
+- Checkpoints are stored in state backend-specific (native) data format (may be incremental depending on the specific backend).
+{{< /hint >}}
+
+Although [savepoints]({{< ref "docs/ops/state/savepoints" >}}) are created internally with the same mechanisms as
+checkpoints, they are conceptually different and can be a bit more expensive to produce and restore from. Their design focuses
+more on portability and operational flexibility, especially with respect to changes to the job.
+The use case for savepoints is for planned, manual operations. For example, this could be an update of your Flink version, changing your job graph, and so on.
+
+{{< hint info >}}
+- Savepoints are created, owned and deleted solely by the user.
+That means, Flink does not delete savepoints neither after job termination nor after
+restore.
+- Savepoints are stored in a state backend independent (canonical) format (Note: Since Flink 1.15, savepoints can be also stored in
+the backend-specific [native]({{< ref "docs/ops/state/savepoints" >}}#savepoint-format) format which is faster to create
+and restore but comes with some limitations.
+{{< /hint >}}
+
+### Capabilities and limitations
+The following table gives an overview of capabilities and limitations for the various types of savepoints and
+checkpoints.
+- ✓ - Flink fully support this type of the snapshot
+- x - Flink doesn't support this type of the snapshot
+- ! - While these operations currently work, Flink doesn't officially guarantee support for them, so there is a certain level of risk associated with them
+
+| Operation | Canonical Savepoint | Native Savepoint | Aligned Checkpoint | Unaligned Checkpoint |
+|:----------------------------------|:--------------------|:-----------------|:-------------------|:---------------------|
+| State backend change | ✓ | x | x | x |
+| State Processor API (writing) | ✓ | x | x | x |
+| State Processor API (reading) | ✓ | ! | ! | x |
+| Self-contained and relocatable | ✓ | ✓ | x | x |
+| Schema evolution | ✓ | ! | ! | ! |
+| Arbitrary job upgrade | ✓ | ✓ | ✓ | x |
+| Non-arbitrary job upgrade | ✓ | ✓ | ✓ | x |
+| Flink minor version upgrade | ✓ | ✓ | ✓ | x |
+| Flink bug/patch version upgrade | ✓ | ✓ | ✓ | ✓ |
+| Rescaling | ✓ | ✓ | ✓ | ✓ |
+
+- [State backend change]({{< ref "docs/ops/state/state_backends" >}}) - configuring a different State Backend than was used when taking the snapshot.
+- [State Processor API (writing)]({{< ref "docs/libs/state_processor_api" >}}#writing-new-savepoints) - the ability to create a new snapshot of this type via the State Processor API.
+- [State Processor API (reading)]({{< ref "docs/libs/state_processor_api" >}}#reading-state) - the ability to read states from an existing snapshot of this type via the State Processor API.
+- Self-contained and relocatable - the one snapshot folder contains everything it needs for recovery
+and it doesn't depend on other snapshots which means it can be easily moved to another place if needed.
+- [Schema evolution]({{< ref "docs/dev/datastream/fault-tolerance/serialization/schema_evolution" >}}) - the *state* data type can be changed if it uses a serializer that supports schema evolution (e.g., POJOs and Avro types)
+- Arbitrary job upgrade - the snapshot can be restored even if the [partitioning types]({{< ref "docs/dev/datastream/operators/overview" >}}#physical-partitioning)(rescale, rebalance, map, etc.)
+or in-flight record types for the existing operators have changed.
+- Non-arbitrary job upgrade - restoring the snapshot is possible with updated operators if the job graph topology and in-flight record types remain unchanged.
+- Flink minor version upgrade - restoring a snapshot taken with an older minor version of Flink (1.x → 1.y).
+- Flink bug/patch version upgrade - restoring a snapshot taken with an older patch version of Flink (1.14.x → 1.14.y).
+- Rescaling - restoring the snapshot with a different parallelism than was used during the snapshot creation.
+
+
+{{< top >}}
diff --git a/docs/content.zh/docs/ops/state/savepoints.md b/docs/content.zh/docs/ops/state/savepoints.md
index 87a2890..11f3368 100644
--- a/docs/content.zh/docs/ops/state/savepoints.md
+++ b/docs/content.zh/docs/ops/state/savepoints.md
@@ -27,7 +27,7 @@ under the License.
# Savepoints
-## 什么是 Savepoint ？ Savepoint 与 Checkpoint 有什么不同？
+## 什么是 Savepoint ？
Savepoint 是依据 Flink [checkpointing 机制]({{< ref "docs/learn-flink/fault_tolerance" >}})所创建的流作业执行状态的一致镜像。 你可以使用 Savepoint 进行 Flink 作业的停止与重启、fork 或者更新。 Savepoint 由两部分组成：稳定存储（列入 HDFS，S3，...) 上包含二进制文件的目录（通常很大），和元数据文件（相对较小）。 稳定存储上的文件表示作业执行状态的数据镜像。 Savepoint 的元数据文件以（相对路径）的形式包含（主要）指向作为 Savepoint 一部分的稳定存储上的所有文件的指针。
@@ -35,11 +35,7 @@ Savepoint 是依据 Flink [checkpointing 机制]({{< ref "docs/learn-flink/fault
**注意:** 为了允许程序和 Flink 版本之间的升级，请务必查看以下有关<a href="#分配算子-id">分配算子 ID </a>的部分 。
{{< /hint >}}
-从概念上讲， Flink 的 Savepoint 与 Checkpoint 的不同之处类似于传统数据库中的备份与恢复日志之间的差异。 Checkpoint 的主要目的是为意外失败的作业提供恢复机制。 Checkpoint 的生命周期由 Flink 管理，即 Flink 创建，管理和删除 Checkpoint - 无需用户交互。 作为一种恢复和定期触发的方法，Checkpoint 实现有两个设计目标：i）轻量级创建和 ii）尽可能快地恢复。 可能会利用某些特定的属性来达到这个，例如， 工作代码在执行尝试之间不会改变。 在用户终止作业后，通常会删除 Checkpoint（除非明确配置为保留的 Checkpoint）。
-
- 与此相反、Savepoint 由用户创建，拥有和删除。 他们的用例是计划的，手动备份和恢复。 例如，升级 Flink 版本，调整用户逻辑，改变并行度，以及进行红蓝部署等。 当然，Savepoint 必须在作业停止后继续存在。 从概念上讲，Savepoint 的生成，恢复成本可能更高一些，Savepoint 更多地关注可移植性和对前面提到的作业更改的支持。
-
-除去这些概念上的差异，Checkpoint 和 Savepoint 的当前实现基本上使用相同的代码并生成相同的格式。然而，目前有一个例外，我们可能会在未来引入更多的差异。例外情况是使用 RocksDB 状态后端的增量 Checkpoint。他们使用了一些 RocksDB 内部格式，而不是 Flink 的本机 Savepoint 格式。这使他们成为了与 Savepoint 相比，更轻量级的 Checkpoint 机制的第一个实例。
+To make proper use of savepoints, it's important to understand the differences between [checkpoints]({{< ref "docs/ops/state/checkpoints" >}}) and savepoints which is described in [checkpoints vs. savepoints]({{< ref "docs/ops/state/checkpoints_vs_savepoints" >}}).
## 分配算子 ID
diff --git a/docs/content/docs/concepts/stateful-stream-processing.md b/docs/content/docs/concepts/stateful-stream-processing.md
index f8c2a77..7f139b0 100644
--- a/docs/content/docs/concepts/stateful-stream-processing.md
+++ b/docs/content/docs/concepts/stateful-stream-processing.md
@@ -312,6 +312,9 @@ mechanism for this.
Savepoints are similar to checkpoints except that they are
**triggered by the user** and **don't automatically expire** when newer
checkpoints are completed.
+To make proper use of savepoints, it's important to understand the differences between
+[checkpoints]({{< ref "docs/ops/state/checkpoints" >}}) and [savepoints]({{< ref "docs/ops/state/savepoints" >}})
+which is described in [checkpoints vs. savepoints]({{< ref "docs/ops/state/checkpoints_vs_savepoints" >}}).
{{< top >}}
diff --git a/docs/content/docs/ops/state/checkpoints.md b/docs/content/docs/ops/state/checkpoints.md
index 63f0c06..2d4f6ec 100644
--- a/docs/content/docs/ops/state/checkpoints.md
+++ b/docs/content/docs/ops/state/checkpoints.md
@@ -35,6 +35,8 @@ the same semantics as a failure-free execution.
See [Checkpointing]({{< ref "docs/dev/datastream/fault-tolerance/checkpointing" >}}) for how to enable and
configure checkpoints for your program.
+To understand the differences between checkpoints and [savepoints]({{< ref "docs/ops/state/savepoints" >}}) see [checkpoints vs. savepoints]({{< ref "docs/ops/state/checkpoints_vs_savepoints" >}}).
+
## Checkpoint Storage
When checkpointing is enabled, managed state is persisted to ensure consistent recovery in case of failures.
@@ -156,12 +158,6 @@ env.getCheckpointConfig().setCheckpointStorage(
new FileSystemCheckpointStorage("hdfs:///checkpoints-data/", FILE_SIZE_THESHOLD));
```
-### Difference to Savepoints
-
-Checkpoints have a few differences from [savepoints]({{< ref "docs/ops/state/savepoints" >}}). They
-- use a state backend specific (low-level) data format, may be incremental. (starting from Flink 1.15 savepoints can also use the backend [native]({{< ref "docs/ops/state/savepoints" >}}#savepoint-format) format.)
-- do not support Flink specific features like rescaling.
-
### Resuming from a retained checkpoint
A job may be resumed from a checkpoint just as from a savepoint
@@ -175,6 +171,4 @@ above).
$ bin/flink run -s :checkpointMetaDataPath [:runArgs]
```
-
-
{{< top >}}
diff --git a/docs/content/docs/ops/state/checkpoints_vs_savepoints.md b/docs/content/docs/ops/state/checkpoints_vs_savepoints.md
new file mode 100644
index 0000000..0b66b25
--- /dev/null
+++ b/docs/content/docs/ops/state/checkpoints_vs_savepoints.md
@@ -0,0 +1,95 @@
+---
+title: "Checkpoints vs. Savepoints"
+weight: 10
+type: docs
+aliases:
+ - /ops/state/checkpoints_vs_savepoints.html
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Checkpoints vs. Savepoints
+
+## Overview
+
+Conceptually, Flink's [savepoints]({{< ref "docs/ops/state/savepoints" >}}) are different from [checkpoints]({{< ref "docs/ops/state/checkpoints" >}})
+in a way that's analogous to how backups are different from recovery logs in traditional database systems.
+
+The primary purpose of checkpoints is to provide a recovery mechanism in case of unexpected job failures.
+A [checkpoint's lifecycle]({{< ref "docs/dev/datastream/fault-tolerance/checkpointing" >}}) is managed by Flink,
+i.e. a checkpoint is created, owned, and released by Flink - without user interaction.
+Because checkpoints are being triggered often, and are relied upon for failure recovery, the two main design goals for the checkpoint implementation are
+i) being as lightweight to create and ii) being as fast to restore from as possible.
+Optimizations towards those goals can exploit certain properties, e.g., that the job code doesn't change between the execution attempts.
+
+{{< hint info >}}
+- Checkpoints are automatically deleted if the application is terminated by the user
+(except if checkpoints are explicitly configured to be retained).
+- Checkpoints are stored in state backend-specific (native) data format (may be incremental depending on the specific backend).
+{{< /hint >}}
+
+Although [savepoints]({{< ref "docs/ops/state/savepoints" >}}) are created internally with the same mechanisms as
+checkpoints, they are conceptually different and can be a bit more expensive to produce and restore from. Their design focuses
+more on portability and operational flexibility, especially with respect to changes to the job.
+The use case for savepoints is for planned, manual operations. For example, this could be an update of your Flink version, changing your job graph, and so on.
+
+{{< hint info >}}
+- Savepoints are created, owned and deleted solely by the user.
+That means, Flink does not delete savepoints neither after job termination nor after
+restore.
+- Savepoints are stored in a state backend independent (canonical) format (Note: Since Flink 1.15, savepoints can be also stored in
+the backend-specific [native]({{< ref "docs/ops/state/savepoints" >}}#savepoint-format) format which is faster to create
+and restore but comes with some limitations.
+{{< /hint >}}
+
+### Capabilities and limitations
+The following table gives an overview of capabilities and limitations for the various types of savepoints and
+checkpoints.
+- ✓ - Flink fully support this type of the snapshot
+- x - Flink doesn't support this type of the snapshot
+- ! - While these operations currently work, Flink doesn't officially guarantee support for them, so there is a certain level of risk associated with them
+
+| Operation | Canonical Savepoint | Native Savepoint | Aligned Checkpoint | Unaligned Checkpoint |
+|:----------------------------------|:--------------------|:-----------------|:-------------------|:---------------------|
+| State backend change | ✓ | x | x | x |
+| State Processor API (writing) | ✓ | x | x | x |
+| State Processor API (reading) | ✓ | ! | ! | x |
+| Self-contained and relocatable | ✓ | ✓ | x | x |
+| Schema evolution | ✓ | ! | ! | ! |
+| Arbitrary job upgrade | ✓ | ✓ | ✓ | x |
+| Non-arbitrary job upgrade | ✓ | ✓ | ✓ | x |
+| Flink minor version upgrade | ✓ | ✓ | ✓ | x |
+| Flink bug/patch version upgrade | ✓ | ✓ | ✓ | ✓ |
+| Rescaling | ✓ | ✓ | ✓ | ✓ |
+
+- [State backend change]({{< ref "docs/ops/state/state_backends" >}}) - configuring a different State Backend than was used when taking the snapshot.
+- [State Processor API (writing)]({{< ref "docs/libs/state_processor_api" >}}#writing-new-savepoints) - the ability to create a new snapshot of this type via the State Processor API.
+- [State Processor API (reading)]({{< ref "docs/libs/state_processor_api" >}}#reading-state) - the ability to read states from an existing snapshot of this type via the State Processor API.
+- Self-contained and relocatable - the one snapshot folder contains everything it needs for recovery
+and it doesn't depend on other snapshots which means it can be easily moved to another place if needed.
+- [Schema evolution]({{< ref "docs/dev/datastream/fault-tolerance/serialization/schema_evolution" >}}) - the *state* data type can be changed if it uses a serializer that supports schema evolution (e.g., POJOs and Avro types)
+- Arbitrary job upgrade - the snapshot can be restored even if the [partitioning types]({{< ref "docs/dev/datastream/operators/overview" >}}#physical-partitioning)(rescale, rebalance, map, etc.)
+or in-flight record types for the existing operators have changed.
+- Non-arbitrary job upgrade - restoring the snapshot is possible with updated operators if the job graph topology and in-flight record types remain unchanged.
+- Flink minor version upgrade - restoring a snapshot taken with an older minor version of Flink (1.x → 1.y).
+- Flink bug/patch version upgrade - restoring a snapshot taken with an older patch version of Flink (1.14.x → 1.14.y).
+- Rescaling - restoring the snapshot with a different parallelism than was used during the snapshot creation.
+
+
+{{< top >}}
diff --git a/docs/content/docs/ops/state/savepoints.md b/docs/content/docs/ops/state/savepoints.md
index a24c4db..2b182e4 100644
--- a/docs/content/docs/ops/state/savepoints.md
+++ b/docs/content/docs/ops/state/savepoints.md
@@ -27,7 +27,7 @@ under the License.
# Savepoints
-## What is a Savepoint? How is a Savepoint different from a Checkpoint?
+## What is a Savepoint?
A Savepoint is a consistent image of the execution state of a streaming job, created via Flink's [checkpointing mechanism]({{< ref "docs/learn-flink/fault_tolerance" >}}). You can use Savepoints to stop-and-resume, fork,
or update your Flink jobs. Savepoints consist of two parts: a directory with (typically large) binary files on stable storage (e.g. HDFS, S3, ...) and a (relatively small) meta data file. The files on stable storage represent the net data of the job's execution state
@@ -37,14 +37,7 @@ image. The meta data file of a Savepoint contains (primarily) pointers to all fi
In order to allow upgrades between programs and Flink versions, it is important to check out the following section about [assigning IDs to your operators](#assigning-operator-ids).
{{< /hint >}}
-Conceptually, Flink's Savepoints are different from Checkpoints in a similar way that backups are different from recovery logs in traditional database systems. The primary purpose of Checkpoints is to provide a recovery mechanism in case of
-unexpected job failures. A Checkpoint's lifecycle is managed by Flink, i.e. a Checkpoint is created, owned, and deleted by Flink - without user interaction. As a method of recovery and being periodically triggered, two main
-design goals for the Checkpoint implementation are i) being as lightweight to create and ii) being as fast to restore from as possible. Optimizations towards those goals can exploit certain properties, e.g. that the job code
-does not change between the execution attempts. Checkpoints are usually dropped after the job was terminated by the user unless the job is explicitly configured to retain checkpoints upon failure or cancellation.
-
-In contrast to all this, Savepoints are created, owned, and deleted by the user. Their use case is planned, manual backup and resume. For example, this could be an update of your Flink version, changing your job graph,
-changing parallelism, forking a second job like for a red/blue deployment, and so on. Of course, Savepoints must survive job termination. Conceptually, Savepoints can be a bit more expensive to produce and restore from and focus
-more on portability and flexibility with respect to changes to the job.
+To make proper use of savepoints, it's important to understand the differences between [checkpoints]({{< ref "docs/ops/state/checkpoints" >}}) and savepoints which is described in [checkpoints vs. savepoints]({{< ref "docs/ops/state/checkpoints_vs_savepoints" >}}).
## Assigning Operator IDs