You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "fredia (via GitHub)" <gi...@apache.org> on 2025/03/18 08:38:09 UTC
[PR] [FLINK-37502][docs] Translate documentation for Disaggregated State Management into Chinese [flink]
fredia opened a new pull request, #26312:
URL: https://github.com/apache/flink/pull/26312
<!--
*Thank you very much for contributing to Apache Flink - we are happy that you want to help us improve Flink. To help the community review your contribution in the best possible way, please go through the checklist below, which will get the contribution into a shape in which it can be best reviewed.*
*Please understand that we do not do this to make contributions to Flink a hassle. In order to uphold a high standard of quality for code contributions, while at the same time managing a large number of contributions, we need contributors to prepare the contributions well, and give reviewers enough contextual information for the review. Please also understand that contributions that do not follow this guide will take longer to review and thus typically be picked up with lower priority by the community.*
## Contribution Checklist
- Make sure that the pull request corresponds to a [JIRA issue](https://issues.apache.org/jira/projects/FLINK/issues). Exceptions are made for typos in JavaDoc or documentation files, which need no JIRA issue.
- Name the pull request in the form "[FLINK-XXXX] [component] Title of the pull request", where *FLINK-XXXX* should be replaced by the actual issue number. Skip *component* if you are unsure about which is the best component.
Typo fixes that have no associated JIRA issue should be named following this pattern: `[hotfix] [docs] Fix typo in event time introduction` or `[hotfix] [javadocs] Expand JavaDoc for PuncuatedWatermarkGenerator`.
- Fill out the template below to describe the changes contributed by the pull request. That will give reviewers the context they need to do the review.
- Make sure that the change passes the automated tests, i.e., `mvn clean verify` passes. You can set up Azure Pipelines CI to do that following [this guide](https://cwiki.apache.org/confluence/display/FLINK/Azure+Pipelines#AzurePipelines-Tutorial:SettingupAzurePipelinesforaforkoftheFlinkrepository).
- Each pull request should address only one issue, not mix up code from multiple issues.
- Each commit in the pull request has a meaningful commit message (including the JIRA id)
- Once all items of the checklist are addressed, remove the above text and this checklist, leaving only the filled out template below.
**(The sections below can be removed for hotfixes of typos)**
-->
## What is the purpose of the change
Chinese documents for Disaggregate State and new State APIs
## Brief change log
- Documents for State API V2
- Documents for ForSt State Backend
- Documents for Disaggregated State
## Verifying this change
This change is a docs work without any test coverage.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
- The S3 file system connector: (no)
## Documentation
- Does this pull request introduce a new feature? (no)
- If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37502][docs] Translate documentation for Disaggregated State Management into Chinese [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #26312:
URL: https://github.com/apache/flink/pull/26312#discussion_r2005923231
##########
docs/content/docs/dev/datastream/fault-tolerance/state_v2.md:
##########
@@ -497,6 +497,8 @@ In a typical stateful Flink Application you don't need operators state. It is
mostly a special type of state that is used in source/sink implementations and
scenarios where you don't have a key by which state can be partitioned.
+**Notes:** Operator state is still not supported in Python DataStream API.
Review Comment:
nit: in Python -> in the Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37502][docs] Translate documentation for Disaggregated State Management into Chinese [flink]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26312:
URL: https://github.com/apache/flink/pull/26312#issuecomment-4333103755
This PR is being marked as stale since it has not had any activity in the last 90 days.
If you would like to keep this PR alive, please leave a comment asking for a review.
If the PR has merge conflicts, update it with the latest from the base branch.
If you are having difficulty finding a reviewer, please reach out to the
community, contact details can be found here: https://flink.apache.org/what-is-flink/community/
If this PR is no longer valid or desired, please feel free to close it.
If no activity occurs in the next 30 days, it will be automatically closed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37502][docs] Translate documentation for Disaggregated State Management into Chinese [flink]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26312:
URL: https://github.com/apache/flink/pull/26312#issuecomment-4561787404
This PR has been closed since it has not had any activity in 120 days.
If you feel like this was a mistake, or you would like to continue working on it,
please feel free to re-open the PR and ask for a review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37502][docs] Translate documentation for Disaggregated State Management into Chinese [flink]
Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #26312:
URL: https://github.com/apache/flink/pull/26312#issuecomment-2732165204
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8711de9d6f9aaa1d61937d6fcc2d1b205e05eb33",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8711de9d6f9aaa1d61937d6fcc2d1b205e05eb33",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8711de9d6f9aaa1d61937d6fcc2d1b205e05eb33 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37502][docs] Translate documentation for Disaggregated State Management into Chinese [flink]
Posted by "Zakelly (via GitHub)" <gi...@apache.org>.
Zakelly commented on code in PR #26312:
URL: https://github.com/apache/flink/pull/26312#discussion_r2020517479
##########
docs/content.zh/docs/dev/datastream/fault-tolerance/state_v2.md:
##########
@@ -25,40 +25,25 @@ specific language governing permissions and limitations
under the License.
-->
-# Working with State V2 (New APIs)
+# 使用 State V2 (New APIs)
+本章节您将了解 Flink 用于编写有状态程序的新 API,要了解有状态流处理背后的概念,请参阅[Stateful Stream
+Processing]({{< ref "docs/concepts/stateful-stream-processing" >}})。
-In this section you will learn about the new APIs that Flink provides for writing
-stateful programs. Please take a look at [Stateful Stream
-Processing]({{< ref "docs/concepts/stateful-stream-processing" >}})
-to learn about the concepts behind stateful stream processing.
-
-The new state API is designed to be more flexible than the previous API. User can perform
-asynchronous state operations, thus making it more powerful and more efficient.
-The asynchronous state access is essential for the state backend to be able to handle
-large state sizes and to be able to spill to remote file systems when necessary.
-This is called the 'disaggregated state management'. For more information about this,
-please see [Disaggregated State Management]({{< ref "docs/ops/state/disaggregated_state" >}}).
+新的 state API 比以前的 API 更灵活,用户可以通过新的 API 使用异步状态操作,这使得新的 API 更加强大有效。
Review Comment:
```suggestion
新的 state API 比以前的 API 更灵活,用户可以通过新的 API 使用异步状态操作,是一套更强大有效的 API。
```
##########
docs/content.zh/docs/dev/datastream/fault-tolerance/state_v2.md:
##########
@@ -25,40 +25,25 @@ specific language governing permissions and limitations
under the License.
-->
-# Working with State V2 (New APIs)
+# 使用 State V2 (New APIs)
+本章节您将了解 Flink 用于编写有状态程序的新 API,要了解有状态流处理背后的概念,请参阅[Stateful Stream
+Processing]({{< ref "docs/concepts/stateful-stream-processing" >}})。
-In this section you will learn about the new APIs that Flink provides for writing
-stateful programs. Please take a look at [Stateful Stream
-Processing]({{< ref "docs/concepts/stateful-stream-processing" >}})
-to learn about the concepts behind stateful stream processing.
-
-The new state API is designed to be more flexible than the previous API. User can perform
-asynchronous state operations, thus making it more powerful and more efficient.
-The asynchronous state access is essential for the state backend to be able to handle
-large state sizes and to be able to spill to remote file systems when necessary.
-This is called the 'disaggregated state management'. For more information about this,
-please see [Disaggregated State Management]({{< ref "docs/ops/state/disaggregated_state" >}}).
+新的 state API 比以前的 API 更灵活,用户可以通过新的 API 使用异步状态操作,这使得新的 API 更加强大有效。
+异步 state 访问对“存算分离”来说是必要的,即令状态后端支持大状态和在必要时将文件溢出到远程文件系统的能力。更多关于
Review Comment:
```suggestion
异步 state 访问对存算分离来说是必要的,即令状态后端支持大状态时将文件溢出到远程文件系统的能力。更多关于
```
##########
docs/content.zh/docs/dev/datastream/fault-tolerance/state_v2.md:
##########
@@ -275,62 +218,42 @@ env.fromElements(Tuple2.of(1L, 3L), Tuple2.of(1L, 5L), Tuple2.of(1L, 7L), Tuple2
{{< /tab >}}
{{< /tabs >}}
-This example implements a poor man's counting window. We key the tuples by the first field
-(in the example all have the same key `1`). The function stores the count and a running sum in
-a `ValueState`. Once the count reaches 2 it will emit the average and clear the state so that
-we start over from `0`. Note that this would keep a different state value for each different input
-key if we had tuples with different values in the first field.
-
-### Execution Order
-
-The state access methods are executed asynchronously. This means that the state access methods
-will not block the current thread. With synchronous APIs, the state access methods will be executed in
-the order they are called. However, with asynchronous APIs, the state access methods will be executed
-out of order, especially for those invokes for different incoming elements. For the above example,
-if the `flatMap` function is invoked for two different incoming elements A and B, the state access
-methods for A and B will be executed. Firstly, `asyncGet` is executed for A, then `asyncGet` is
-allowed to execute for B. The finish order of the two `asyncGet` is not guaranteed. Thus the order
-of continuation of the two `StateFuture`s is not guaranteed. Thus invokes of `asyncClear` or
-`asyncUpdate`for A and B are not determined.
-
-Although the state access methods are executed out of order, this not mean that all the user code
-are run in parallel. The user code in the `processElement`, `flatMap` or `thenXXxx` methods
-following the state access methods will be executed in a single thread (the task thread). So there
-is no concurrency issue for the user code.
-
-Typically, you don't need to worry about the execution order of the state access methods, but there
-is still some rules the Flink will ensure:
-* The execution order of user code entry `flatMap` for same-key elements are invoked strictly in
-order of element arrival.
-* The consumers or functions passed to the `thenXXxx` methods are executed in the order they are
-chained. If they are not chained, or there are multiple chains, the order is not guaranteed.
-
-### Best practice of asynchronous APIs
-
-The asynchronous APIs are designed to be more efficient and more powerful than the synchronous APIs.
-There are some best practices that you should follow when using the asynchronous APIs:
-* **Avoid mixing synchronous and asynchronous state access**
-* Use chaining of `thenXXxx` methods to handle the result of the state access and then gives another
- state access or result. Divide the logic into multiple steps split by `thenXXxx` methods.
-* Avoid accessing mutable members of the user function (`RichFlatMapFunction`). Since the state access
- methods are executed out of order, the mutable members may be accessed in unpredictable order.
- Instead, use the result of the state access to pass the data between different steps. The
- `StateFutureUtils.completedFuture` or `thenApply` method can be used to pass the data. Or use
- a captured container (`AtomicReference`) which is initialized for each invoke of the `flatMap`
- to share between lambdas.
-
-
-### State Time-To-Live (TTL)
-
-A *time-to-live* (TTL) can be assigned to the keyed state of any type. If a TTL is configured and a
-state value has expired, the stored value will be cleaned up on a best effort basis which is
-discussed in more detail below.
-
-All state collection types support per-entry TTLs. This means that list elements and map entries
-expire independently.
-
-In order to use state TTL one must first build a `StateTtlConfig` configuration object. The TTL
-functionality can then be enabled in any state descriptor by passing the configuration:
+这个例子实现了一个简单的计数窗口。 我们把元组的第一个元素当作 key(在示例中都 key 都是 "1")。
+该函数将出现的次数以及总和存储在 "ValueState" 中。
+一旦出现次数达到 2,则将平均值发送到下游,并清除状态重新开始。 请注意,我们会为每个不同的 key(元组中第一个元素)保存一个单独的值。
+
+### 执行顺序
+状态访问是异步的,这意味着状态访问方法不会阻塞当前线程。在同步 API 下,状态访问方法会按照它们被调用的顺序执行,但是异步 API 下,
+尤其是对于不同输入元素的状态访问方法,状态访问方法可能会乱序执行。在上面的例子中,如果 `flatMap` 函数接收搭配两个不同的
+输入元素 A 和 B, A 和 B 的状态访问方法会被执行。首先,元素 A 的 `asyncGet`方法被执行,然后元素 B 的 `asyncGet` 方法接着被执行,
+但两个`asyncGet`的完成顺序不能保证。因此,无法保证两个 `StateFuture` 后续步骤的执行顺序,即无法确定元素 A 和 元素 B 的
Review Comment:
```suggestion
但两个`asyncGet`的完成顺序不能保证。因此我们无法保证两个 `StateFuture` 后续步骤的执行顺序,也就无法确定元素 A 和 元素 B 的
```
##########
docs/content.zh/docs/dev/datastream/fault-tolerance/state_v2.md:
##########
@@ -79,143 +64,101 @@ KeyedStream<WordCount> keyed = words
{{< top >}}
-## Using Keyed State V2
-
-Unlike the previous state API, the new state API is designed for asynchronous state access.
-Each type of state gives two versions of the API: synchronous and asynchronous. The synchronous
-API is a blocking API that waits for the state access to complete. The asynchronous API is
-non-blocking and returns a `StateFuture` that will be completed when the state access
-is done. After that a callback or following logic will be invoked (if any).
-The asynchronous API is more efficient and should be used whenever possible.
-It is highly **not** recommended to mixed synchronous and asynchronous state access in the same
-user function.
-
-The keyed state interfaces provide access to different types of state that are all scoped to
-the key of the current input element. This means that this type of state can only be used
-on a `KeyedStream`, which can be created via `stream.keyBy(…)` in Java. And then most importantly,
-the keyed stream needs to be enabled for asynchronous state access by calling `enableAsyncState()`.
-The new API set are only available on `KeyedStream` with `enableAsyncState()` invoked.
-
-Now, we will look at the different types of state available, and then we will see
-how they can be used in a program. Since the synchronous APIs are identical with the original APIs,
-We only focus on the asynchronous ones here.
-
-### The Return Values
-
-First of all, we should get familiar with the return value of those asynchronous state access methods.
-
-`StateFuture<T>` is a future that will be completed with the result of the state access.
-The return bype is T. It provides multiple methods to handle the result, listed as:
-* `StateFuture<Void> thenAccept(Consumer<T>)`: This method takes a `Consumer` that will be called with the result
- when the state access is done. It returns a `StateFuture<Void>`, which will be finished when the
- `Consumer` is done.
-* `StateFuture<R> thenApply(Function<T, R>)`: This method takes a `Function` that will be called with the result
- when the state access is done. The return value of the function will be the result of the following
- `StateFuture`, which will be finished when the `Function` is done.
-* `StateFuture<R> thenCompose(Function<T, StateFuture<R>>)`: This method takes a `Function` that will
- be called with the result when the state access is done. The return value of the function should be
- a `StateFuture<R>`, which is exposed to the invoker of `thenCompose` as a return value.
- The `StateFuture<R>` will be finished when the inner `StateFuture<R>` of `Function` is finished.
-* `StateFuture<R> thenCombine(StateFuture<U>, BiFunction<T, U, R>)`: This method takes another `StateFuture` and a
- `BiFunction` that will be called with the results of both `StateFuture`s when they are done. The return
- value of the `BiFunction` will be the result of the following `StateFuture`, which will be finished when
- the `BiFunction` is done.
-
-Those methods are similar to the corresponding ones of the `CompletableFuture`. Besides these methods,
-keep in mind that `StateFuture` does not provide a `get()` method to block the current thread until the
-state access is done. This is because blocking the current thread may cause recursive blocking. The
-`StateFuture<T>` also provides conditional version of `thenAccept`, `thenApply`, `thenCompose` and `thenCombine`,
-which is for the case that the state access is done and the following logic will be split into two branches
-based on the result of the state access. The conditional version of those methods are `thenConditionallyAccept`,
-`thenConditionallyApply`, `thenConditionallyCompose` and `thenConditionallyCombine`.
-
-`StateIterator<T>` is an iterator that can be used to iterate over the elements of a state. It provides
-the following methods:
-* `boolean isEmpty()` : A synchronous method returns a `true` if the iterator has no elements, and `false` otherwise.
-* `StateFuture<Void> onNext(Consumer<T>)` : This method takes a `Consumer` that will be called with the next element
- when the state access is done. It returns a `StateFuture<Void>`, which will be finished when the `Consumer` is done.
- Also, a function version of `onNext` is provided, which is `StateFuture<Collection<R>> onNext(Function<T, R>)`. This method
- takes a `Function` that will be called with the next element when the state access is done. The return value of the function
- will be collected and returned as a collection of the following `StateFuture`, which will be
- finished when the `Function` is done.
-
-We also provide a `StateFutureUtils` class that contains some utility methods to handle `StateFuture`s.
-These methods are:
-* `StateFuture<T> completedFuture(T)`: This method returns a completed `StateFuture` with the given value. This is
-useful when you want to return a constant value in a `thenCompose` method for further processing.
-* `StateFuture<Void> completedVoidFuture()`: This method returns a completed `StateFuture` with `null` value.
-A void value version of `completedFuture`.
-* `StateFuture<Collection<T>> combineAll(Collection<StateFuture<T>>)` : This method takes a collection of `StateFuture`s
-and returns a `StateFuture` that will be completed when all the input `StateFuture`s are completed. The result of the
-returned `StateFuture` is a collection of the results of the input `StateFuture`s. This is useful when you want to
-combine the results of multiple `StateFuture`s.
-* `StateFuture<Iterable<T>> toIterable(StateFuture<StateIterator<T>>)` : This method takes a `StateFuture` of `StateIterator`
-and returns a `StateFuture` of `Iterable`. The result of the returned `StateFuture` is an `Iterable` that contains all
-the elements of the `StateIterator`. This is useful when you want to convert a `StateIterator` to an `Iterable`.
-There is no good reason to do so, since this may disable the capability of lazy loading. Only useful when the further
-calculation depends on the whole data from the iterator.
-
-
-### State Primitives
-
-The available state primitives are:
-
-* `ValueState<T>`: This keeps a value that can be updated and
-retrieved (scoped to key of the input element as mentioned above, so there will possibly be one value
-for each key that the operation sees). The value can be set using `asyncUpdate(T)` and retrieved using
-`StateFuture<T> asyncValue()`.
-
-* `ListState<T>`: This keeps a list of elements. You can append elements and retrieve an `StateIterator`
-over all currently stored elements. Elements are added using `asyncAdd(T)` or `asyncAddAll(List<T>)`,
-the Iterable can be retrieved using `StateFuture<StateIterator<T>> asyncGet()`.
-You can also override the existing list with `asyncUpdate(List<T>)`
-
-* `ReducingState<T>`: This keeps a single value that represents the aggregation of all values
-added to the state. The interface is similar to `ListState` but elements added using
-`asyncAdd(T)` are reduced to an aggregate using a specified `ReduceFunction`.
-
-* `AggregatingState<IN, OUT>`: This keeps a single value that represents the aggregation of all values
-added to the state. Contrary to `ReducingState`, the aggregate type may be different from the type
-of elements that are added to the state. The interface is the same as for `ListState` but elements
-added using `asyncAdd(IN)` are aggregated using a specified `AggregateFunction`.
-
-* `MapState<UK, UV>`: This keeps a list of mappings. You can put key-value pairs into the state and
-retrieve an `StateIterator` over all currently stored mappings. Mappings are added using `asyncPut(UK, UV)` or
-`asyncPutAll(Map<UK, UV>)`. The value associated with a user key can be retrieved using `asyncGet(UK)`.
-The iterable views for mappings, keys and values can be retrieved using `asyncEntries()`,
-`asyncKeys()` and `asyncValues()` respectively. You can also use `asyncIsEmpty()` to check whether
-this map contains any key-value mappings.
-
-All types of state also have a method `asyncClear()` that clears the state for the currently
-active key, i.e. the key of the input element.
-
-It is important to keep in mind that these state objects are only used for interfacing
-with state. The state is not necessarily stored inside but might reside on disk or somewhere else.
-The second thing to keep in mind is that the value you get from the state
-depends on the key of the input element. So the value you get in one invocation of your
-user function can differ from the value in another invocation if the keys involved are different.
-
-To get a state handle, you have to create a `StateDescriptor`. This holds the name of the state
-(as we will see later, you can create several states, and they have to have unique names so
-that you can reference them), the type of the values that the state holds, and possibly
-a user-specified function, such as a `ReduceFunction`. Depending on what type of state you
-want to retrieve, you create either a `ValueStateDescriptor`, a `ListStateDescriptor`,
-an `AggregatingStateDescriptor`, a `ReducingStateDescriptor`, or a `MapStateDescriptor`.
-To differentiate between the previous state APIs, you should use the `StateDescriptor`s under the
-`org.apache.flink.api.common.state.v2` package (note the **v2**).
-
-State is accessed using the `RuntimeContext`, so it is only possible in *rich functions*.
-Please see [here]({{< ref "docs/dev/datastream/user_defined_functions" >}}#rich-functions) for
-information about that, but we will also see an example shortly. The `RuntimeContext` that
-is available in a `RichFunction` has these methods for accessing state:
+## 使用 Keyed State V2
+
+和之前的 state API 不同,新的 state API 允许用户使用异步状态访问。每种类型的状态都提供了两种版本的 API:同步和异步。
+同步 API 是一个阻塞等待状态访问完成的 API。异步 API 是非阻塞的,它会返回一个 `StateFuture`,这个 `StateFuture` 会在状态访问完成时完成。
+在 `StateFuture` 完成时,callback 或者 后续逻辑将会被调用(如果有的话)。异步 API 比同步 API 更加高效,更推荐使用异步 API。
Review Comment:
```suggestion
在 `StateFuture` 完成时,用户的回调会被执行。异步 API 比同步 API 更加高效,我们更推荐使用异步 API。
```
##########
docs/content.zh/docs/dev/datastream/fault-tolerance/state_v2.md:
##########
@@ -275,62 +218,42 @@ env.fromElements(Tuple2.of(1L, 3L), Tuple2.of(1L, 5L), Tuple2.of(1L, 7L), Tuple2
{{< /tab >}}
{{< /tabs >}}
-This example implements a poor man's counting window. We key the tuples by the first field
-(in the example all have the same key `1`). The function stores the count and a running sum in
-a `ValueState`. Once the count reaches 2 it will emit the average and clear the state so that
-we start over from `0`. Note that this would keep a different state value for each different input
-key if we had tuples with different values in the first field.
-
-### Execution Order
-
-The state access methods are executed asynchronously. This means that the state access methods
-will not block the current thread. With synchronous APIs, the state access methods will be executed in
-the order they are called. However, with asynchronous APIs, the state access methods will be executed
-out of order, especially for those invokes for different incoming elements. For the above example,
-if the `flatMap` function is invoked for two different incoming elements A and B, the state access
-methods for A and B will be executed. Firstly, `asyncGet` is executed for A, then `asyncGet` is
-allowed to execute for B. The finish order of the two `asyncGet` is not guaranteed. Thus the order
-of continuation of the two `StateFuture`s is not guaranteed. Thus invokes of `asyncClear` or
-`asyncUpdate`for A and B are not determined.
-
-Although the state access methods are executed out of order, this not mean that all the user code
-are run in parallel. The user code in the `processElement`, `flatMap` or `thenXXxx` methods
-following the state access methods will be executed in a single thread (the task thread). So there
-is no concurrency issue for the user code.
-
-Typically, you don't need to worry about the execution order of the state access methods, but there
-is still some rules the Flink will ensure:
-* The execution order of user code entry `flatMap` for same-key elements are invoked strictly in
-order of element arrival.
-* The consumers or functions passed to the `thenXXxx` methods are executed in the order they are
-chained. If they are not chained, or there are multiple chains, the order is not guaranteed.
-
-### Best practice of asynchronous APIs
-
-The asynchronous APIs are designed to be more efficient and more powerful than the synchronous APIs.
-There are some best practices that you should follow when using the asynchronous APIs:
-* **Avoid mixing synchronous and asynchronous state access**
-* Use chaining of `thenXXxx` methods to handle the result of the state access and then gives another
- state access or result. Divide the logic into multiple steps split by `thenXXxx` methods.
-* Avoid accessing mutable members of the user function (`RichFlatMapFunction`). Since the state access
- methods are executed out of order, the mutable members may be accessed in unpredictable order.
- Instead, use the result of the state access to pass the data between different steps. The
- `StateFutureUtils.completedFuture` or `thenApply` method can be used to pass the data. Or use
- a captured container (`AtomicReference`) which is initialized for each invoke of the `flatMap`
- to share between lambdas.
-
-
-### State Time-To-Live (TTL)
-
-A *time-to-live* (TTL) can be assigned to the keyed state of any type. If a TTL is configured and a
-state value has expired, the stored value will be cleaned up on a best effort basis which is
-discussed in more detail below.
-
-All state collection types support per-entry TTLs. This means that list elements and map entries
-expire independently.
-
-In order to use state TTL one must first build a `StateTtlConfig` configuration object. The TTL
-functionality can then be enabled in any state descriptor by passing the configuration:
+这个例子实现了一个简单的计数窗口。 我们把元组的第一个元素当作 key(在示例中都 key 都是 "1")。
+该函数将出现的次数以及总和存储在 "ValueState" 中。
+一旦出现次数达到 2,则将平均值发送到下游,并清除状态重新开始。 请注意,我们会为每个不同的 key(元组中第一个元素)保存一个单独的值。
+
+### 执行顺序
+状态访问是异步的,这意味着状态访问方法不会阻塞当前线程。在同步 API 下,状态访问方法会按照它们被调用的顺序执行,但是异步 API 下,
+尤其是对于不同输入元素的状态访问方法,状态访问方法可能会乱序执行。在上面的例子中,如果 `flatMap` 函数接收搭配两个不同的
+输入元素 A 和 B, A 和 B 的状态访问方法会被执行。首先,元素 A 的 `asyncGet`方法被执行,然后元素 B 的 `asyncGet` 方法接着被执行,
+但两个`asyncGet`的完成顺序不能保证。因此,无法保证两个 `StateFuture` 后续步骤的执行顺序,即无法确定元素 A 和 元素 B 的
+`asyncClear` 和 `asyncUpdate` 的调用顺序。
+
+尽管状态访问方法可能会乱序执行,这也不意味着所有用户代码会串行执行。在 `processElement`, `flatMap` 或者 `thenXXxx`
Review Comment:
```suggestion
尽管状态访问方法可能会乱序执行,这也不意味着所有用户代码会并行执行。在 `processElement`, `flatMap` 或者 `thenXXxx`
```
##########
docs/content.zh/docs/dev/datastream/fault-tolerance/state_v2.md:
##########
@@ -25,40 +25,25 @@ specific language governing permissions and limitations
under the License.
-->
-# Working with State V2 (New APIs)
+# 使用 State V2 (New APIs)
Review Comment:
```suggestion
# 使用 State V2 (新的 API)
```
##########
docs/content.zh/docs/dev/datastream/fault-tolerance/state_v2.md:
##########
@@ -79,143 +64,101 @@ KeyedStream<WordCount> keyed = words
{{< top >}}
-## Using Keyed State V2
-
-Unlike the previous state API, the new state API is designed for asynchronous state access.
-Each type of state gives two versions of the API: synchronous and asynchronous. The synchronous
-API is a blocking API that waits for the state access to complete. The asynchronous API is
-non-blocking and returns a `StateFuture` that will be completed when the state access
-is done. After that a callback or following logic will be invoked (if any).
-The asynchronous API is more efficient and should be used whenever possible.
-It is highly **not** recommended to mixed synchronous and asynchronous state access in the same
-user function.
-
-The keyed state interfaces provide access to different types of state that are all scoped to
-the key of the current input element. This means that this type of state can only be used
-on a `KeyedStream`, which can be created via `stream.keyBy(…)` in Java. And then most importantly,
-the keyed stream needs to be enabled for asynchronous state access by calling `enableAsyncState()`.
-The new API set are only available on `KeyedStream` with `enableAsyncState()` invoked.
-
-Now, we will look at the different types of state available, and then we will see
-how they can be used in a program. Since the synchronous APIs are identical with the original APIs,
-We only focus on the asynchronous ones here.
-
-### The Return Values
-
-First of all, we should get familiar with the return value of those asynchronous state access methods.
-
-`StateFuture<T>` is a future that will be completed with the result of the state access.
-The return bype is T. It provides multiple methods to handle the result, listed as:
-* `StateFuture<Void> thenAccept(Consumer<T>)`: This method takes a `Consumer` that will be called with the result
- when the state access is done. It returns a `StateFuture<Void>`, which will be finished when the
- `Consumer` is done.
-* `StateFuture<R> thenApply(Function<T, R>)`: This method takes a `Function` that will be called with the result
- when the state access is done. The return value of the function will be the result of the following
- `StateFuture`, which will be finished when the `Function` is done.
-* `StateFuture<R> thenCompose(Function<T, StateFuture<R>>)`: This method takes a `Function` that will
- be called with the result when the state access is done. The return value of the function should be
- a `StateFuture<R>`, which is exposed to the invoker of `thenCompose` as a return value.
- The `StateFuture<R>` will be finished when the inner `StateFuture<R>` of `Function` is finished.
-* `StateFuture<R> thenCombine(StateFuture<U>, BiFunction<T, U, R>)`: This method takes another `StateFuture` and a
- `BiFunction` that will be called with the results of both `StateFuture`s when they are done. The return
- value of the `BiFunction` will be the result of the following `StateFuture`, which will be finished when
- the `BiFunction` is done.
-
-Those methods are similar to the corresponding ones of the `CompletableFuture`. Besides these methods,
-keep in mind that `StateFuture` does not provide a `get()` method to block the current thread until the
-state access is done. This is because blocking the current thread may cause recursive blocking. The
-`StateFuture<T>` also provides conditional version of `thenAccept`, `thenApply`, `thenCompose` and `thenCombine`,
-which is for the case that the state access is done and the following logic will be split into two branches
-based on the result of the state access. The conditional version of those methods are `thenConditionallyAccept`,
-`thenConditionallyApply`, `thenConditionallyCompose` and `thenConditionallyCombine`.
-
-`StateIterator<T>` is an iterator that can be used to iterate over the elements of a state. It provides
-the following methods:
-* `boolean isEmpty()` : A synchronous method returns a `true` if the iterator has no elements, and `false` otherwise.
-* `StateFuture<Void> onNext(Consumer<T>)` : This method takes a `Consumer` that will be called with the next element
- when the state access is done. It returns a `StateFuture<Void>`, which will be finished when the `Consumer` is done.
- Also, a function version of `onNext` is provided, which is `StateFuture<Collection<R>> onNext(Function<T, R>)`. This method
- takes a `Function` that will be called with the next element when the state access is done. The return value of the function
- will be collected and returned as a collection of the following `StateFuture`, which will be
- finished when the `Function` is done.
-
-We also provide a `StateFutureUtils` class that contains some utility methods to handle `StateFuture`s.
-These methods are:
-* `StateFuture<T> completedFuture(T)`: This method returns a completed `StateFuture` with the given value. This is
-useful when you want to return a constant value in a `thenCompose` method for further processing.
-* `StateFuture<Void> completedVoidFuture()`: This method returns a completed `StateFuture` with `null` value.
-A void value version of `completedFuture`.
-* `StateFuture<Collection<T>> combineAll(Collection<StateFuture<T>>)` : This method takes a collection of `StateFuture`s
-and returns a `StateFuture` that will be completed when all the input `StateFuture`s are completed. The result of the
-returned `StateFuture` is a collection of the results of the input `StateFuture`s. This is useful when you want to
-combine the results of multiple `StateFuture`s.
-* `StateFuture<Iterable<T>> toIterable(StateFuture<StateIterator<T>>)` : This method takes a `StateFuture` of `StateIterator`
-and returns a `StateFuture` of `Iterable`. The result of the returned `StateFuture` is an `Iterable` that contains all
-the elements of the `StateIterator`. This is useful when you want to convert a `StateIterator` to an `Iterable`.
-There is no good reason to do so, since this may disable the capability of lazy loading. Only useful when the further
-calculation depends on the whole data from the iterator.
-
-
-### State Primitives
-
-The available state primitives are:
-
-* `ValueState<T>`: This keeps a value that can be updated and
-retrieved (scoped to key of the input element as mentioned above, so there will possibly be one value
-for each key that the operation sees). The value can be set using `asyncUpdate(T)` and retrieved using
-`StateFuture<T> asyncValue()`.
-
-* `ListState<T>`: This keeps a list of elements. You can append elements and retrieve an `StateIterator`
-over all currently stored elements. Elements are added using `asyncAdd(T)` or `asyncAddAll(List<T>)`,
-the Iterable can be retrieved using `StateFuture<StateIterator<T>> asyncGet()`.
-You can also override the existing list with `asyncUpdate(List<T>)`
-
-* `ReducingState<T>`: This keeps a single value that represents the aggregation of all values
-added to the state. The interface is similar to `ListState` but elements added using
-`asyncAdd(T)` are reduced to an aggregate using a specified `ReduceFunction`.
-
-* `AggregatingState<IN, OUT>`: This keeps a single value that represents the aggregation of all values
-added to the state. Contrary to `ReducingState`, the aggregate type may be different from the type
-of elements that are added to the state. The interface is the same as for `ListState` but elements
-added using `asyncAdd(IN)` are aggregated using a specified `AggregateFunction`.
-
-* `MapState<UK, UV>`: This keeps a list of mappings. You can put key-value pairs into the state and
-retrieve an `StateIterator` over all currently stored mappings. Mappings are added using `asyncPut(UK, UV)` or
-`asyncPutAll(Map<UK, UV>)`. The value associated with a user key can be retrieved using `asyncGet(UK)`.
-The iterable views for mappings, keys and values can be retrieved using `asyncEntries()`,
-`asyncKeys()` and `asyncValues()` respectively. You can also use `asyncIsEmpty()` to check whether
-this map contains any key-value mappings.
-
-All types of state also have a method `asyncClear()` that clears the state for the currently
-active key, i.e. the key of the input element.
-
-It is important to keep in mind that these state objects are only used for interfacing
-with state. The state is not necessarily stored inside but might reside on disk or somewhere else.
-The second thing to keep in mind is that the value you get from the state
-depends on the key of the input element. So the value you get in one invocation of your
-user function can differ from the value in another invocation if the keys involved are different.
-
-To get a state handle, you have to create a `StateDescriptor`. This holds the name of the state
-(as we will see later, you can create several states, and they have to have unique names so
-that you can reference them), the type of the values that the state holds, and possibly
-a user-specified function, such as a `ReduceFunction`. Depending on what type of state you
-want to retrieve, you create either a `ValueStateDescriptor`, a `ListStateDescriptor`,
-an `AggregatingStateDescriptor`, a `ReducingStateDescriptor`, or a `MapStateDescriptor`.
-To differentiate between the previous state APIs, you should use the `StateDescriptor`s under the
-`org.apache.flink.api.common.state.v2` package (note the **v2**).
-
-State is accessed using the `RuntimeContext`, so it is only possible in *rich functions*.
-Please see [here]({{< ref "docs/dev/datastream/user_defined_functions" >}}#rich-functions) for
-information about that, but we will also see an example shortly. The `RuntimeContext` that
-is available in a `RichFunction` has these methods for accessing state:
+## 使用 Keyed State V2
+
+和之前的 state API 不同,新的 state API 允许用户使用异步状态访问。每种类型的状态都提供了两种版本的 API:同步和异步。
+同步 API 是一个阻塞等待状态访问完成的 API。异步 API 是非阻塞的,它会返回一个 `StateFuture`,这个 `StateFuture` 会在状态访问完成时完成。
+在 `StateFuture` 完成时,callback 或者 后续逻辑将会被调用(如果有的话)。异步 API 比同步 API 更加高效,更推荐使用异步 API。
+注意,**不建议**在同一个用户函数中混合使用同步和异步状态访问。
Review Comment:
```suggestion
需要注意,**不建议**在同一个用户函数中混合使用同步和异步状态访问。
```
##########
docs/content.zh/docs/dev/datastream/fault-tolerance/state_v2.md:
##########
@@ -275,62 +218,42 @@ env.fromElements(Tuple2.of(1L, 3L), Tuple2.of(1L, 5L), Tuple2.of(1L, 7L), Tuple2
{{< /tab >}}
{{< /tabs >}}
-This example implements a poor man's counting window. We key the tuples by the first field
-(in the example all have the same key `1`). The function stores the count and a running sum in
-a `ValueState`. Once the count reaches 2 it will emit the average and clear the state so that
-we start over from `0`. Note that this would keep a different state value for each different input
-key if we had tuples with different values in the first field.
-
-### Execution Order
-
-The state access methods are executed asynchronously. This means that the state access methods
-will not block the current thread. With synchronous APIs, the state access methods will be executed in
-the order they are called. However, with asynchronous APIs, the state access methods will be executed
-out of order, especially for those invokes for different incoming elements. For the above example,
-if the `flatMap` function is invoked for two different incoming elements A and B, the state access
-methods for A and B will be executed. Firstly, `asyncGet` is executed for A, then `asyncGet` is
-allowed to execute for B. The finish order of the two `asyncGet` is not guaranteed. Thus the order
-of continuation of the two `StateFuture`s is not guaranteed. Thus invokes of `asyncClear` or
-`asyncUpdate`for A and B are not determined.
-
-Although the state access methods are executed out of order, this not mean that all the user code
-are run in parallel. The user code in the `processElement`, `flatMap` or `thenXXxx` methods
-following the state access methods will be executed in a single thread (the task thread). So there
-is no concurrency issue for the user code.
-
-Typically, you don't need to worry about the execution order of the state access methods, but there
-is still some rules the Flink will ensure:
-* The execution order of user code entry `flatMap` for same-key elements are invoked strictly in
-order of element arrival.
-* The consumers or functions passed to the `thenXXxx` methods are executed in the order they are
-chained. If they are not chained, or there are multiple chains, the order is not guaranteed.
-
-### Best practice of asynchronous APIs
-
-The asynchronous APIs are designed to be more efficient and more powerful than the synchronous APIs.
-There are some best practices that you should follow when using the asynchronous APIs:
-* **Avoid mixing synchronous and asynchronous state access**
-* Use chaining of `thenXXxx` methods to handle the result of the state access and then gives another
- state access or result. Divide the logic into multiple steps split by `thenXXxx` methods.
-* Avoid accessing mutable members of the user function (`RichFlatMapFunction`). Since the state access
- methods are executed out of order, the mutable members may be accessed in unpredictable order.
- Instead, use the result of the state access to pass the data between different steps. The
- `StateFutureUtils.completedFuture` or `thenApply` method can be used to pass the data. Or use
- a captured container (`AtomicReference`) which is initialized for each invoke of the `flatMap`
- to share between lambdas.
-
-
-### State Time-To-Live (TTL)
-
-A *time-to-live* (TTL) can be assigned to the keyed state of any type. If a TTL is configured and a
-state value has expired, the stored value will be cleaned up on a best effort basis which is
-discussed in more detail below.
-
-All state collection types support per-entry TTLs. This means that list elements and map entries
-expire independently.
-
-In order to use state TTL one must first build a `StateTtlConfig` configuration object. The TTL
-functionality can then be enabled in any state descriptor by passing the configuration:
+这个例子实现了一个简单的计数窗口。 我们把元组的第一个元素当作 key(在示例中都 key 都是 "1")。
+该函数将出现的次数以及总和存储在 "ValueState" 中。
+一旦出现次数达到 2,则将平均值发送到下游,并清除状态重新开始。 请注意,我们会为每个不同的 key(元组中第一个元素)保存一个单独的值。
+
+### 执行顺序
+状态访问是异步的,这意味着状态访问方法不会阻塞当前线程。在同步 API 下,状态访问方法会按照它们被调用的顺序执行,但是异步 API 下,
+尤其是对于不同输入元素的状态访问方法,状态访问方法可能会乱序执行。在上面的例子中,如果 `flatMap` 函数接收搭配两个不同的
+输入元素 A 和 B, A 和 B 的状态访问方法会被执行。首先,元素 A 的 `asyncGet`方法被执行,然后元素 B 的 `asyncGet` 方法接着被执行,
+但两个`asyncGet`的完成顺序不能保证。因此,无法保证两个 `StateFuture` 后续步骤的执行顺序,即无法确定元素 A 和 元素 B 的
+`asyncClear` 和 `asyncUpdate` 的调用顺序。
+
+尽管状态访问方法可能会乱序执行,这也不意味着所有用户代码会串行执行。在 `processElement`, `flatMap` 或者 `thenXXxx`
+方法中的用户代码都会在单个线程(任务线程)中执行。因此不会出现用户代码的并发问题。
Review Comment:
```suggestion
方法中的用户代码都会在单个线程(即主任务线程)中执行。因此不会出现用户代码的并发问题。
```
##########
docs/content.zh/docs/dev/datastream/fault-tolerance/state_v2.md:
##########
@@ -275,62 +218,42 @@ env.fromElements(Tuple2.of(1L, 3L), Tuple2.of(1L, 5L), Tuple2.of(1L, 7L), Tuple2
{{< /tab >}}
{{< /tabs >}}
-This example implements a poor man's counting window. We key the tuples by the first field
-(in the example all have the same key `1`). The function stores the count and a running sum in
-a `ValueState`. Once the count reaches 2 it will emit the average and clear the state so that
-we start over from `0`. Note that this would keep a different state value for each different input
-key if we had tuples with different values in the first field.
-
-### Execution Order
-
-The state access methods are executed asynchronously. This means that the state access methods
-will not block the current thread. With synchronous APIs, the state access methods will be executed in
-the order they are called. However, with asynchronous APIs, the state access methods will be executed
-out of order, especially for those invokes for different incoming elements. For the above example,
-if the `flatMap` function is invoked for two different incoming elements A and B, the state access
-methods for A and B will be executed. Firstly, `asyncGet` is executed for A, then `asyncGet` is
-allowed to execute for B. The finish order of the two `asyncGet` is not guaranteed. Thus the order
-of continuation of the two `StateFuture`s is not guaranteed. Thus invokes of `asyncClear` or
-`asyncUpdate`for A and B are not determined.
-
-Although the state access methods are executed out of order, this not mean that all the user code
-are run in parallel. The user code in the `processElement`, `flatMap` or `thenXXxx` methods
-following the state access methods will be executed in a single thread (the task thread). So there
-is no concurrency issue for the user code.
-
-Typically, you don't need to worry about the execution order of the state access methods, but there
-is still some rules the Flink will ensure:
-* The execution order of user code entry `flatMap` for same-key elements are invoked strictly in
-order of element arrival.
-* The consumers or functions passed to the `thenXXxx` methods are executed in the order they are
-chained. If they are not chained, or there are multiple chains, the order is not guaranteed.
-
-### Best practice of asynchronous APIs
-
-The asynchronous APIs are designed to be more efficient and more powerful than the synchronous APIs.
-There are some best practices that you should follow when using the asynchronous APIs:
-* **Avoid mixing synchronous and asynchronous state access**
-* Use chaining of `thenXXxx` methods to handle the result of the state access and then gives another
- state access or result. Divide the logic into multiple steps split by `thenXXxx` methods.
-* Avoid accessing mutable members of the user function (`RichFlatMapFunction`). Since the state access
- methods are executed out of order, the mutable members may be accessed in unpredictable order.
- Instead, use the result of the state access to pass the data between different steps. The
- `StateFutureUtils.completedFuture` or `thenApply` method can be used to pass the data. Or use
- a captured container (`AtomicReference`) which is initialized for each invoke of the `flatMap`
- to share between lambdas.
-
-
-### State Time-To-Live (TTL)
-
-A *time-to-live* (TTL) can be assigned to the keyed state of any type. If a TTL is configured and a
-state value has expired, the stored value will be cleaned up on a best effort basis which is
-discussed in more detail below.
-
-All state collection types support per-entry TTLs. This means that list elements and map entries
-expire independently.
-
-In order to use state TTL one must first build a `StateTtlConfig` configuration object. The TTL
-functionality can then be enabled in any state descriptor by passing the configuration:
+这个例子实现了一个简单的计数窗口。 我们把元组的第一个元素当作 key(在示例中都 key 都是 "1")。
+该函数将出现的次数以及总和存储在 "ValueState" 中。
+一旦出现次数达到 2,则将平均值发送到下游,并清除状态重新开始。 请注意,我们会为每个不同的 key(元组中第一个元素)保存一个单独的值。
+
+### 执行顺序
+状态访问是异步的,这意味着状态访问方法不会阻塞当前线程。在同步 API 下,状态访问方法会按照它们被调用的顺序执行,但是异步 API 下,
+尤其是对于不同输入元素的状态访问方法,状态访问方法可能会乱序执行。在上面的例子中,如果 `flatMap` 函数接收搭配两个不同的
+输入元素 A 和 B, A 和 B 的状态访问方法会被执行。首先,元素 A 的 `asyncGet`方法被执行,然后元素 B 的 `asyncGet` 方法接着被执行,
+但两个`asyncGet`的完成顺序不能保证。因此,无法保证两个 `StateFuture` 后续步骤的执行顺序,即无法确定元素 A 和 元素 B 的
+`asyncClear` 和 `asyncUpdate` 的调用顺序。
+
+尽管状态访问方法可能会乱序执行,这也不意味着所有用户代码会串行执行。在 `processElement`, `flatMap` 或者 `thenXXxx`
+方法中的用户代码都会在单个线程(任务线程)中执行。因此不会出现用户代码的并发问题。
+
+通常,您无需担心状态访问方法的执行顺序问题,但 Flink 仍确保一些规则:
+* 同 key 的元素的执行顺序严格按照元素到达 `flatMap` 的顺序执行的。
+* 传递给 `thenXXxx` 方法的函数会按照调用链的顺序执行。如果它们不属于调用链,或者同时有多个调用链,则无法保证执行顺序。
Review Comment:
```suggestion
* 传递给 `thenXXxx` 方法的函数会按照调用链的顺序执行。如果同时有多个调用链,则无法保证执行顺序。
```
##########
docs/content.zh/docs/dev/datastream/fault-tolerance/state_v2.md:
##########
@@ -275,62 +218,42 @@ env.fromElements(Tuple2.of(1L, 3L), Tuple2.of(1L, 5L), Tuple2.of(1L, 7L), Tuple2
{{< /tab >}}
{{< /tabs >}}
-This example implements a poor man's counting window. We key the tuples by the first field
-(in the example all have the same key `1`). The function stores the count and a running sum in
-a `ValueState`. Once the count reaches 2 it will emit the average and clear the state so that
-we start over from `0`. Note that this would keep a different state value for each different input
-key if we had tuples with different values in the first field.
-
-### Execution Order
-
-The state access methods are executed asynchronously. This means that the state access methods
-will not block the current thread. With synchronous APIs, the state access methods will be executed in
-the order they are called. However, with asynchronous APIs, the state access methods will be executed
-out of order, especially for those invokes for different incoming elements. For the above example,
-if the `flatMap` function is invoked for two different incoming elements A and B, the state access
-methods for A and B will be executed. Firstly, `asyncGet` is executed for A, then `asyncGet` is
-allowed to execute for B. The finish order of the two `asyncGet` is not guaranteed. Thus the order
-of continuation of the two `StateFuture`s is not guaranteed. Thus invokes of `asyncClear` or
-`asyncUpdate`for A and B are not determined.
-
-Although the state access methods are executed out of order, this not mean that all the user code
-are run in parallel. The user code in the `processElement`, `flatMap` or `thenXXxx` methods
-following the state access methods will be executed in a single thread (the task thread). So there
-is no concurrency issue for the user code.
-
-Typically, you don't need to worry about the execution order of the state access methods, but there
-is still some rules the Flink will ensure:
-* The execution order of user code entry `flatMap` for same-key elements are invoked strictly in
-order of element arrival.
-* The consumers or functions passed to the `thenXXxx` methods are executed in the order they are
-chained. If they are not chained, or there are multiple chains, the order is not guaranteed.
-
-### Best practice of asynchronous APIs
-
-The asynchronous APIs are designed to be more efficient and more powerful than the synchronous APIs.
-There are some best practices that you should follow when using the asynchronous APIs:
-* **Avoid mixing synchronous and asynchronous state access**
-* Use chaining of `thenXXxx` methods to handle the result of the state access and then gives another
- state access or result. Divide the logic into multiple steps split by `thenXXxx` methods.
-* Avoid accessing mutable members of the user function (`RichFlatMapFunction`). Since the state access
- methods are executed out of order, the mutable members may be accessed in unpredictable order.
- Instead, use the result of the state access to pass the data between different steps. The
- `StateFutureUtils.completedFuture` or `thenApply` method can be used to pass the data. Or use
- a captured container (`AtomicReference`) which is initialized for each invoke of the `flatMap`
- to share between lambdas.
-
-
-### State Time-To-Live (TTL)
-
-A *time-to-live* (TTL) can be assigned to the keyed state of any type. If a TTL is configured and a
-state value has expired, the stored value will be cleaned up on a best effort basis which is
-discussed in more detail below.
-
-All state collection types support per-entry TTLs. This means that list elements and map entries
-expire independently.
-
-In order to use state TTL one must first build a `StateTtlConfig` configuration object. The TTL
-functionality can then be enabled in any state descriptor by passing the configuration:
+这个例子实现了一个简单的计数窗口。 我们把元组的第一个元素当作 key(在示例中都 key 都是 "1")。
+该函数将出现的次数以及总和存储在 "ValueState" 中。
+一旦出现次数达到 2,则将平均值发送到下游,并清除状态重新开始。 请注意,我们会为每个不同的 key(元组中第一个元素)保存一个单独的值。
+
+### 执行顺序
+状态访问是异步的,这意味着状态访问方法不会阻塞当前线程。在同步 API 下,状态访问方法会按照它们被调用的顺序执行,但是异步 API 下,
+尤其是对于不同输入元素的状态访问方法,状态访问方法可能会乱序执行。在上面的例子中,如果 `flatMap` 函数接收搭配两个不同的
+输入元素 A 和 B, A 和 B 的状态访问方法会被执行。首先,元素 A 的 `asyncGet`方法被执行,然后元素 B 的 `asyncGet` 方法接着被执行,
+但两个`asyncGet`的完成顺序不能保证。因此,无法保证两个 `StateFuture` 后续步骤的执行顺序,即无法确定元素 A 和 元素 B 的
+`asyncClear` 和 `asyncUpdate` 的调用顺序。
+
+尽管状态访问方法可能会乱序执行,这也不意味着所有用户代码会串行执行。在 `processElement`, `flatMap` 或者 `thenXXxx`
+方法中的用户代码都会在单个线程(任务线程)中执行。因此不会出现用户代码的并发问题。
+
+通常,您无需担心状态访问方法的执行顺序问题,但 Flink 仍确保一些规则:
+* 同 key 的元素的执行顺序严格按照元素到达 `flatMap` 的顺序执行的。
+* 传递给 `thenXXxx` 方法的函数会按照调用链的顺序执行。如果它们不属于调用链,或者同时有多个调用链,则无法保证执行顺序。
+
+### 异步 APIs 的最佳实践
+
+异步 APIs 的设计比同步 APIs 更加高效和强大。当使用异步 APIs 时,有一些最佳实践应该遵循:
+* **不要混合同步和异步状态访问**
+* 使用 `thenXXxx` 方法的链式调用来处理状态访问的结果,然后给出另一个状态访问或结果。将逻辑分割成多个步骤,通过 `thenXXxx`
+ 方法分割。
Review Comment:
* 将逻辑分割成多个步骤,通过连续的 `thenXXxx` 来进行链式调用,前一个 `thenXXxx` 的结果传递给后一个 `thenXXxx` 。
##########
docs/content.zh/docs/dev/datastream/fault-tolerance/state_v2.md:
##########
@@ -275,62 +218,42 @@ env.fromElements(Tuple2.of(1L, 3L), Tuple2.of(1L, 5L), Tuple2.of(1L, 7L), Tuple2
{{< /tab >}}
{{< /tabs >}}
-This example implements a poor man's counting window. We key the tuples by the first field
-(in the example all have the same key `1`). The function stores the count and a running sum in
-a `ValueState`. Once the count reaches 2 it will emit the average and clear the state so that
-we start over from `0`. Note that this would keep a different state value for each different input
-key if we had tuples with different values in the first field.
-
-### Execution Order
-
-The state access methods are executed asynchronously. This means that the state access methods
-will not block the current thread. With synchronous APIs, the state access methods will be executed in
-the order they are called. However, with asynchronous APIs, the state access methods will be executed
-out of order, especially for those invokes for different incoming elements. For the above example,
-if the `flatMap` function is invoked for two different incoming elements A and B, the state access
-methods for A and B will be executed. Firstly, `asyncGet` is executed for A, then `asyncGet` is
-allowed to execute for B. The finish order of the two `asyncGet` is not guaranteed. Thus the order
-of continuation of the two `StateFuture`s is not guaranteed. Thus invokes of `asyncClear` or
-`asyncUpdate`for A and B are not determined.
-
-Although the state access methods are executed out of order, this not mean that all the user code
-are run in parallel. The user code in the `processElement`, `flatMap` or `thenXXxx` methods
-following the state access methods will be executed in a single thread (the task thread). So there
-is no concurrency issue for the user code.
-
-Typically, you don't need to worry about the execution order of the state access methods, but there
-is still some rules the Flink will ensure:
-* The execution order of user code entry `flatMap` for same-key elements are invoked strictly in
-order of element arrival.
-* The consumers or functions passed to the `thenXXxx` methods are executed in the order they are
-chained. If they are not chained, or there are multiple chains, the order is not guaranteed.
-
-### Best practice of asynchronous APIs
-
-The asynchronous APIs are designed to be more efficient and more powerful than the synchronous APIs.
-There are some best practices that you should follow when using the asynchronous APIs:
-* **Avoid mixing synchronous and asynchronous state access**
-* Use chaining of `thenXXxx` methods to handle the result of the state access and then gives another
- state access or result. Divide the logic into multiple steps split by `thenXXxx` methods.
-* Avoid accessing mutable members of the user function (`RichFlatMapFunction`). Since the state access
- methods are executed out of order, the mutable members may be accessed in unpredictable order.
- Instead, use the result of the state access to pass the data between different steps. The
- `StateFutureUtils.completedFuture` or `thenApply` method can be used to pass the data. Or use
- a captured container (`AtomicReference`) which is initialized for each invoke of the `flatMap`
- to share between lambdas.
-
-
-### State Time-To-Live (TTL)
-
-A *time-to-live* (TTL) can be assigned to the keyed state of any type. If a TTL is configured and a
-state value has expired, the stored value will be cleaned up on a best effort basis which is
-discussed in more detail below.
-
-All state collection types support per-entry TTLs. This means that list elements and map entries
-expire independently.
-
-In order to use state TTL one must first build a `StateTtlConfig` configuration object. The TTL
-functionality can then be enabled in any state descriptor by passing the configuration:
+这个例子实现了一个简单的计数窗口。 我们把元组的第一个元素当作 key(在示例中都 key 都是 "1")。
+该函数将出现的次数以及总和存储在 "ValueState" 中。
+一旦出现次数达到 2,则将平均值发送到下游,并清除状态重新开始。 请注意,我们会为每个不同的 key(元组中第一个元素)保存一个单独的值。
+
+### 执行顺序
+状态访问是异步的,这意味着状态访问方法不会阻塞当前线程。在同步 API 下,状态访问方法会按照它们被调用的顺序执行,但是异步 API 下,
+尤其是对于不同输入元素的状态访问方法,状态访问方法可能会乱序执行。在上面的例子中,如果 `flatMap` 函数接收搭配两个不同的
+输入元素 A 和 B, A 和 B 的状态访问方法会被执行。首先,元素 A 的 `asyncGet`方法被执行,然后元素 B 的 `asyncGet` 方法接着被执行,
+但两个`asyncGet`的完成顺序不能保证。因此,无法保证两个 `StateFuture` 后续步骤的执行顺序,即无法确定元素 A 和 元素 B 的
+`asyncClear` 和 `asyncUpdate` 的调用顺序。
+
+尽管状态访问方法可能会乱序执行,这也不意味着所有用户代码会串行执行。在 `processElement`, `flatMap` 或者 `thenXXxx`
+方法中的用户代码都会在单个线程(任务线程)中执行。因此不会出现用户代码的并发问题。
+
+通常,您无需担心状态访问方法的执行顺序问题,但 Flink 仍确保一些规则:
+* 同 key 的元素的执行顺序严格按照元素到达 `flatMap` 的顺序执行的。
+* 传递给 `thenXXxx` 方法的函数会按照调用链的顺序执行。如果它们不属于调用链,或者同时有多个调用链,则无法保证执行顺序。
+
+### 异步 APIs 的最佳实践
+
+异步 APIs 的设计比同步 APIs 更加高效和强大。当使用异步 APIs 时,有一些最佳实践应该遵循:
+* **不要混合同步和异步状态访问**
+* 使用 `thenXXxx` 方法的链式调用来处理状态访问的结果,然后给出另一个状态访问或结果。将逻辑分割成多个步骤,通过 `thenXXxx`
+ 方法分割。
+* 避免在用户函数(例如 `RichFlatMapFunction`) 中使用可变(mutable)成员。因为状态访问方法是乱序执行的,
+ 可变成员可能会以无法预测的顺序执行。相反,使用 state access 的结果来在不同步骤之间传递数据。`StateFutureUtils.completedFuture`
+ 和 `thenApply` 方法可以被用来传递数据。或者一个捕获变量的容器 (`AtomicReference`),该容器在每次调用 `flatMap`时
+ 都会初始化,以便在 lambda 之间共享。
Review Comment:
```suggestion
* 避免在用户函数(例如 `RichFlatMapFunction`) 中使用可变(mutable)成员。因为状态访问方法是乱序执行的,
可变成员可能会以无法预测的顺序执行。相反,使用 state access 的结果来在不同步骤之间传递数据。`StateFutureUtils.completedFuture`
和 `thenApply` 方法可以被用来传递数据。或者使用一个每次在 `flatMap` 调用时初始化的容器 (`AtomicReference`),以便在不同回调之间共享数据。
```
##########
docs/content.zh/docs/dev/datastream/fault-tolerance/state_v2.md:
##########
@@ -79,143 +64,101 @@ KeyedStream<WordCount> keyed = words
{{< top >}}
-## Using Keyed State V2
-
-Unlike the previous state API, the new state API is designed for asynchronous state access.
-Each type of state gives two versions of the API: synchronous and asynchronous. The synchronous
-API is a blocking API that waits for the state access to complete. The asynchronous API is
-non-blocking and returns a `StateFuture` that will be completed when the state access
-is done. After that a callback or following logic will be invoked (if any).
-The asynchronous API is more efficient and should be used whenever possible.
-It is highly **not** recommended to mixed synchronous and asynchronous state access in the same
-user function.
-
-The keyed state interfaces provide access to different types of state that are all scoped to
-the key of the current input element. This means that this type of state can only be used
-on a `KeyedStream`, which can be created via `stream.keyBy(…)` in Java. And then most importantly,
-the keyed stream needs to be enabled for asynchronous state access by calling `enableAsyncState()`.
-The new API set are only available on `KeyedStream` with `enableAsyncState()` invoked.
-
-Now, we will look at the different types of state available, and then we will see
-how they can be used in a program. Since the synchronous APIs are identical with the original APIs,
-We only focus on the asynchronous ones here.
-
-### The Return Values
-
-First of all, we should get familiar with the return value of those asynchronous state access methods.
-
-`StateFuture<T>` is a future that will be completed with the result of the state access.
-The return bype is T. It provides multiple methods to handle the result, listed as:
-* `StateFuture<Void> thenAccept(Consumer<T>)`: This method takes a `Consumer` that will be called with the result
- when the state access is done. It returns a `StateFuture<Void>`, which will be finished when the
- `Consumer` is done.
-* `StateFuture<R> thenApply(Function<T, R>)`: This method takes a `Function` that will be called with the result
- when the state access is done. The return value of the function will be the result of the following
- `StateFuture`, which will be finished when the `Function` is done.
-* `StateFuture<R> thenCompose(Function<T, StateFuture<R>>)`: This method takes a `Function` that will
- be called with the result when the state access is done. The return value of the function should be
- a `StateFuture<R>`, which is exposed to the invoker of `thenCompose` as a return value.
- The `StateFuture<R>` will be finished when the inner `StateFuture<R>` of `Function` is finished.
-* `StateFuture<R> thenCombine(StateFuture<U>, BiFunction<T, U, R>)`: This method takes another `StateFuture` and a
- `BiFunction` that will be called with the results of both `StateFuture`s when they are done. The return
- value of the `BiFunction` will be the result of the following `StateFuture`, which will be finished when
- the `BiFunction` is done.
-
-Those methods are similar to the corresponding ones of the `CompletableFuture`. Besides these methods,
-keep in mind that `StateFuture` does not provide a `get()` method to block the current thread until the
-state access is done. This is because blocking the current thread may cause recursive blocking. The
-`StateFuture<T>` also provides conditional version of `thenAccept`, `thenApply`, `thenCompose` and `thenCombine`,
-which is for the case that the state access is done and the following logic will be split into two branches
-based on the result of the state access. The conditional version of those methods are `thenConditionallyAccept`,
-`thenConditionallyApply`, `thenConditionallyCompose` and `thenConditionallyCombine`.
-
-`StateIterator<T>` is an iterator that can be used to iterate over the elements of a state. It provides
-the following methods:
-* `boolean isEmpty()` : A synchronous method returns a `true` if the iterator has no elements, and `false` otherwise.
-* `StateFuture<Void> onNext(Consumer<T>)` : This method takes a `Consumer` that will be called with the next element
- when the state access is done. It returns a `StateFuture<Void>`, which will be finished when the `Consumer` is done.
- Also, a function version of `onNext` is provided, which is `StateFuture<Collection<R>> onNext(Function<T, R>)`. This method
- takes a `Function` that will be called with the next element when the state access is done. The return value of the function
- will be collected and returned as a collection of the following `StateFuture`, which will be
- finished when the `Function` is done.
-
-We also provide a `StateFutureUtils` class that contains some utility methods to handle `StateFuture`s.
-These methods are:
-* `StateFuture<T> completedFuture(T)`: This method returns a completed `StateFuture` with the given value. This is
-useful when you want to return a constant value in a `thenCompose` method for further processing.
-* `StateFuture<Void> completedVoidFuture()`: This method returns a completed `StateFuture` with `null` value.
-A void value version of `completedFuture`.
-* `StateFuture<Collection<T>> combineAll(Collection<StateFuture<T>>)` : This method takes a collection of `StateFuture`s
-and returns a `StateFuture` that will be completed when all the input `StateFuture`s are completed. The result of the
-returned `StateFuture` is a collection of the results of the input `StateFuture`s. This is useful when you want to
-combine the results of multiple `StateFuture`s.
-* `StateFuture<Iterable<T>> toIterable(StateFuture<StateIterator<T>>)` : This method takes a `StateFuture` of `StateIterator`
-and returns a `StateFuture` of `Iterable`. The result of the returned `StateFuture` is an `Iterable` that contains all
-the elements of the `StateIterator`. This is useful when you want to convert a `StateIterator` to an `Iterable`.
-There is no good reason to do so, since this may disable the capability of lazy loading. Only useful when the further
-calculation depends on the whole data from the iterator.
-
-
-### State Primitives
-
-The available state primitives are:
-
-* `ValueState<T>`: This keeps a value that can be updated and
-retrieved (scoped to key of the input element as mentioned above, so there will possibly be one value
-for each key that the operation sees). The value can be set using `asyncUpdate(T)` and retrieved using
-`StateFuture<T> asyncValue()`.
-
-* `ListState<T>`: This keeps a list of elements. You can append elements and retrieve an `StateIterator`
-over all currently stored elements. Elements are added using `asyncAdd(T)` or `asyncAddAll(List<T>)`,
-the Iterable can be retrieved using `StateFuture<StateIterator<T>> asyncGet()`.
-You can also override the existing list with `asyncUpdate(List<T>)`
-
-* `ReducingState<T>`: This keeps a single value that represents the aggregation of all values
-added to the state. The interface is similar to `ListState` but elements added using
-`asyncAdd(T)` are reduced to an aggregate using a specified `ReduceFunction`.
-
-* `AggregatingState<IN, OUT>`: This keeps a single value that represents the aggregation of all values
-added to the state. Contrary to `ReducingState`, the aggregate type may be different from the type
-of elements that are added to the state. The interface is the same as for `ListState` but elements
-added using `asyncAdd(IN)` are aggregated using a specified `AggregateFunction`.
-
-* `MapState<UK, UV>`: This keeps a list of mappings. You can put key-value pairs into the state and
-retrieve an `StateIterator` over all currently stored mappings. Mappings are added using `asyncPut(UK, UV)` or
-`asyncPutAll(Map<UK, UV>)`. The value associated with a user key can be retrieved using `asyncGet(UK)`.
-The iterable views for mappings, keys and values can be retrieved using `asyncEntries()`,
-`asyncKeys()` and `asyncValues()` respectively. You can also use `asyncIsEmpty()` to check whether
-this map contains any key-value mappings.
-
-All types of state also have a method `asyncClear()` that clears the state for the currently
-active key, i.e. the key of the input element.
-
-It is important to keep in mind that these state objects are only used for interfacing
-with state. The state is not necessarily stored inside but might reside on disk or somewhere else.
-The second thing to keep in mind is that the value you get from the state
-depends on the key of the input element. So the value you get in one invocation of your
-user function can differ from the value in another invocation if the keys involved are different.
-
-To get a state handle, you have to create a `StateDescriptor`. This holds the name of the state
-(as we will see later, you can create several states, and they have to have unique names so
-that you can reference them), the type of the values that the state holds, and possibly
-a user-specified function, such as a `ReduceFunction`. Depending on what type of state you
-want to retrieve, you create either a `ValueStateDescriptor`, a `ListStateDescriptor`,
-an `AggregatingStateDescriptor`, a `ReducingStateDescriptor`, or a `MapStateDescriptor`.
-To differentiate between the previous state APIs, you should use the `StateDescriptor`s under the
-`org.apache.flink.api.common.state.v2` package (note the **v2**).
-
-State is accessed using the `RuntimeContext`, so it is only possible in *rich functions*.
-Please see [here]({{< ref "docs/dev/datastream/user_defined_functions" >}}#rich-functions) for
-information about that, but we will also see an example shortly. The `RuntimeContext` that
-is available in a `RichFunction` has these methods for accessing state:
+## 使用 Keyed State V2
+
+和之前的 state API 不同,新的 state API 允许用户使用异步状态访问。每种类型的状态都提供了两种版本的 API:同步和异步。
+同步 API 是一个阻塞等待状态访问完成的 API。异步 API 是非阻塞的,它会返回一个 `StateFuture`,这个 `StateFuture` 会在状态访问完成时完成。
+在 `StateFuture` 完成时,callback 或者 后续逻辑将会被调用(如果有的话)。异步 API 比同步 API 更加高效,更推荐使用异步 API。
+注意,**不建议**在同一个用户函数中混合使用同步和异步状态访问。
+
+Key state 接口提供了对不同类型状态的访问,它们都是以 key 为作用域。这意味着这个类型的状态只能在 `KeyedStream` 上使用,`keyedStream`
+可以通过 `stream.keyBy(…)` 在 Java 中创建。最重要的是,`keyedStream` 需要通过调用 `enableAsyncState()` 来开启异步状态访问。
+新的 API 集合只有在 `KeyedStream` 上开启异步状态访问时才可用。
+
+接下来,我们将看到不同类型的状态,并看到它们如何在程序中使用。由于同步 API 与之前的 API 是等价的,这里只关注异步 API。
+
+### 返回值
+
+首先,我们应该熟悉异步状态访问方法的返回值。
+`StateFuture<T>` 完成时会返回状态访问的结果。返回类型是 T。它提供了多个方法来处理结果,包括:
+* `StateFuture<Void> thenAccept(Consumer<T>)`: 接受一个在状态访问完成时会被调用的 `Consumer` 作为参数。
+ 它返回一个 `StateFuture<Void>`,在 `Consumer` 执行完时完成。
+* `StateFuture<R> thenApply(Function<T, R>)`: 接受一个在状态访问完成时会被调用的 `Function` 作为参数。
+ 返回值 `StateFuture<R>` 会使用 `Function` 的返回值会作为内部结果,在 `Function` 执行完时完成。
+* `StateFuture<R> thenCompose(Function<T, StateFuture<R>>)`: 接受一个在状态访问完成时会被调用的
+ `Function` 作为参数。 函数的返回值是 `StateFuture<R>`, 被传给 `thenCompose` 的调用者作为最终的返回值。返回值
+ `StateFuture<R>` 会在 `Function` 内部的 `StateFuture<R>` 完成时完成。
+* `StateFuture<R> thenCombine(StateFuture<U>, BiFunction<T, U, R>)`: 接受另一个 `StateFuture<U>`
+ 和一个 `BiFunction` 作为参数,`BiFunction` 在两个 `StateFuture` 都完成时被调用。 返回值 `StateFuture` 使用
+ `BiFunction` 的返回值作为内部结果, 返回值的 `StateFuture<R>` 会在两个 `StateFuture` 都完成时完成。
+
+以上这些方法类似于 `CompletableFuture` 的相关方法。在此之外,注意,`StateFuture` 不提供 `get()` 方法来阻塞当前线程,
+直到状态访问完成。 这是因为阻塞当前线程可能会导致递归阻塞。 `StateFuture<T>` 还提供了条件版本的 `thenAccept`,
+`thenApply`, `thenCompose` 和 `thenCombine`,用于后续逻辑依赖 状态访问返回值来选择不同分支的情景。这些方法的条件版本是:
+`thenConditionallyAccept`, `thenConditionallyApply`, `thenConditionallyCompose` 和 `thenConditionallyCombine`。
+
+`StateIterator<T>` 是一个迭代器,可用来迭代状态中的元素。它提供了以下方法:
+* `boolean isEmpty()` : 当迭代器没有元素时返回 `true`,否则返回 `false`,是一个同步方法。
+* `StateFuture<Void> onNext(Consumer<T>)` : 该方法接受一个 `Consumer`作为参数,`Consumer` 会在状态访问完成时与下一个
+ 元素一起被调用。它返回一个 `StateFuture<Void>`,在 `Consumer` 执行完时完成。同样,`onNext` 也提供了 `Function`
+ 作为参数的版本:`StateFuture<Collection<R>> onNext(Function<T, R>)`。该方法接受一个 `Function` 作为参数,
+ 会在状态访问完成时与下一个元素一起被调用。`Function` 的返回值会被收集到一个集合中作为返回值 `StateFuture`的内部
+ 结果,返回值在 `Function` 执行完时完成。
+
+我们也提供了 `StateFutureUtils` 类,其中包含一些用于处理 `StateFuture` 的实用方法。这些方法是:
+* `StateFuture<T> completedFuture(T)`: 该方法返回一个已经完成的 `StateFuture`,该 `StateFuture` 的返回值为给定的值。
+* `StateFuture<Void> completedVoidFuture()`: 该方法返回一个已经完成的 `StateFuture`,该 `StateFuture` 的返回值为 `null`,
+ 是 `completedFuture` 的 void 返回值版本。
+* `StateFuture<Collection<T>> combineAll(Collection<StateFuture<T>>)` : 该方法接受一个 `StateFuture` 集合
+ 作为参数,返回一个 `StateFuture`,返回值 `StateFuture` 在所有的 `StateFuture` 都完成后完成。返回值 `StateFuture`
+ 的内部结果是输入的 `StateFuture` 的结果的集合。该方法在想组合多个 `StateFuture` 时的结果时非常有用。
+* `StateFuture<Iterable<T>> toIterable(StateFuture<StateIterator<T>>)` : 该方法接受一个结果为 `StateIterator`
+ 的`StateFuture` 作为参数,返回一个 `Iterable` 作为结果的 `StateFuture`。 返回值 `StateFuture` 的内部结果
+ `Iterable` 包含了 `StateIterator` 的所有元素。该函数在想把 `StateIterator` 转换为 `Iterable` 时非常有用。
+ 但没有充分的理由这样做,因为这可能会禁用延迟加载的功能。只有当进一步的计算取决于迭代器的整个数据时才有用。
Review Comment:
```suggestion
但这可能会禁用延迟加载的功能,如果没有充分理由这样做,建议不这样使用。本方法只有当下一步的计算依赖于迭代器的全部数据时才有用。
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37502][docs] Translate documentation for Disaggregated State Management into Chinese [flink]
Posted by "Zakelly (via GitHub)" <gi...@apache.org>.
Zakelly commented on code in PR #26312:
URL: https://github.com/apache/flink/pull/26312#discussion_r2022536387
##########
docs/content.zh/docs/ops/state/disaggregated_state.md:
##########
@@ -25,58 +25,43 @@ specific language governing permissions and limitations
under the License.
-->
-# Disaggregated State Management
-
-## Overview
-
-For the first ten years of Flink, the state management is based on memory or local disk of the TaskManager.
-This approach works well for most use cases, but it has some limitations:
- * **Local Disk Constraints**: The state size is limited by the memory or disk size of the TaskManager.
- * **Spiky Resource Usage**: The local state model triggers periodic CPU and network I/O bursts during checkpointing or SST files compaction.
- * **Heavy Recovery**: State needs to be downloaded during recovery. The recovery time is
-proportional to the state size, which can be slow for large state sizes.
-
-In Flink 2.0, we introduced the disaggregated state management. This feature allows users to store
-the state in external storage systems like S3, HDFS, etc. This is useful when the state size
-is extremely large. It could be used to store the state in a more cost-effective way, or to
-persist or recovery the state in a more lightweight way. The benefits of disaggregated state management are:
- * **Unlimited State Size**: The state size is only limited by the external storage system.
- * **Stable Resource Usage**: The state is stored in external storage, thus the checkpoint could be very lightweight.
-And the SST files compaction could be done remotely (TODO).
- * **Fast Recovery**: No need to download the state during recovery. The recovery time is
-independent of the state size.
- * **Flexible**: Users can easily choose different external storage systems or I/O performance levels,
-or scale the storage based on their requirements without change their hardware.
- * **Cost-effective**: External storage are usually cheaper than local disk. Users can flexibly
-adjust computing resources and storage resources independently if there is any bottleneck.
-
-The disaggregated state management contains three parts:
- * **ForSt State Backend**: A state backend that stores the state in external storage systems. It
-can also leverage the local disk for caching and buffering. The asynchronous I/O model is used to
-read and write the state. For more details, see [ForSt State Backend]({{< ref "docs/ops/state/state_backends#the-forststatebackend" >}}).
- * **New State APIs**: The new state APIs (State V2) are introduced to perform asynchronous state
-reads and writes, which is essential for overcoming the high network latency when accessing
-the disaggregated state. For more details, see [New State APIs]({{< ref "docs/dev/datastream/fault-tolerance/state_v2" >}}).
- * **SQL Support**: Many SQL operators are rewritten to support the disaggregated state management
-and asynchronous state access. User can easily enable these by setting the configuration.
+# 存算分离(Disaggregated State Management)
+
+## 概览
+
+在 Flink 的前 10 年,状态管理是基于 TaskManager 的内存或本地磁盘。该方法适用于大多数用户场景,但存在一些限制:
+* **本地磁盘限制**: 状态大小受限于 TaskManager 的内存或者磁盘大小。
+* **资源使用尖刺**:本地状态模型会在 checkpoint 时触发 SST 文件的 compaction,引起 CPU 和网络 I/O 的尖刺。
+* **恢复繁重**:在恢复期间,需要下载状态。恢复时间与状态大小成比例,大状态的恢复非常慢。
Review Comment:
```suggestion
* **恢复很慢**:在恢复期间,状态后端需要下载状态。恢复时间与状态大小成比例,大状态的恢复非常慢。
```
##########
docs/content.zh/docs/ops/state/disaggregated_state.md:
##########
@@ -25,58 +25,43 @@ specific language governing permissions and limitations
under the License.
-->
-# Disaggregated State Management
-
-## Overview
-
-For the first ten years of Flink, the state management is based on memory or local disk of the TaskManager.
-This approach works well for most use cases, but it has some limitations:
- * **Local Disk Constraints**: The state size is limited by the memory or disk size of the TaskManager.
- * **Spiky Resource Usage**: The local state model triggers periodic CPU and network I/O bursts during checkpointing or SST files compaction.
- * **Heavy Recovery**: State needs to be downloaded during recovery. The recovery time is
-proportional to the state size, which can be slow for large state sizes.
-
-In Flink 2.0, we introduced the disaggregated state management. This feature allows users to store
-the state in external storage systems like S3, HDFS, etc. This is useful when the state size
-is extremely large. It could be used to store the state in a more cost-effective way, or to
-persist or recovery the state in a more lightweight way. The benefits of disaggregated state management are:
- * **Unlimited State Size**: The state size is only limited by the external storage system.
- * **Stable Resource Usage**: The state is stored in external storage, thus the checkpoint could be very lightweight.
-And the SST files compaction could be done remotely (TODO).
- * **Fast Recovery**: No need to download the state during recovery. The recovery time is
-independent of the state size.
- * **Flexible**: Users can easily choose different external storage systems or I/O performance levels,
-or scale the storage based on their requirements without change their hardware.
- * **Cost-effective**: External storage are usually cheaper than local disk. Users can flexibly
-adjust computing resources and storage resources independently if there is any bottleneck.
-
-The disaggregated state management contains three parts:
- * **ForSt State Backend**: A state backend that stores the state in external storage systems. It
-can also leverage the local disk for caching and buffering. The asynchronous I/O model is used to
-read and write the state. For more details, see [ForSt State Backend]({{< ref "docs/ops/state/state_backends#the-forststatebackend" >}}).
- * **New State APIs**: The new state APIs (State V2) are introduced to perform asynchronous state
-reads and writes, which is essential for overcoming the high network latency when accessing
-the disaggregated state. For more details, see [New State APIs]({{< ref "docs/dev/datastream/fault-tolerance/state_v2" >}}).
- * **SQL Support**: Many SQL operators are rewritten to support the disaggregated state management
-and asynchronous state access. User can easily enable these by setting the configuration.
+# 存算分离(Disaggregated State Management)
+
+## 概览
+
+在 Flink 的前 10 年,状态管理是基于 TaskManager 的内存或本地磁盘。该方法适用于大多数用户场景,但存在一些限制:
+* **本地磁盘限制**: 状态大小受限于 TaskManager 的内存或者磁盘大小。
+* **资源使用尖刺**:本地状态模型会在 checkpoint 时触发 SST 文件的 compaction,引起 CPU 和网络 I/O 的尖刺。
+* **恢复繁重**:在恢复期间,需要下载状态。恢复时间与状态大小成比例,大状态的恢复非常慢。
+
+在 Flink 2.0,我们引入了存算分离(disaggregated state management). 存算分离允许用户将状态直接存储在第三方存储系统,如 S3,HDFS 等,
+这在状态非常大时很有用。存算分离以一种更有效的方式存储状态,轻量地持久化或恢复状态。存算分离有如下好处:
+* **状态大小无限制**:状态大小仅取决于外部存储系统大小。
+* **资源使用稳定**:状态被存储在外部存储上,因此 checkpoint 可以非常轻量。SST 文件的 compaction 也可以被放在远程(TODO)。
+* **恢复快速**:在恢复期间无需下载状态到本地,恢复时间与状态大小无关。
+* **灵活**:用户可以轻松地选择不同的外部存储系统或I/O性能级别,或者根据需求独立地扩展存储资源,而不需要改变他们的硬件。
+* **经济实惠**:外部存储通常比本地磁盘便宜。如果有瓶颈,用户可以灵活、独立地调整计算资源以及存储资源。
+
+存算分离包括三个部分:
+* **ForSt 状态后端**: 一个以外部存储系统作为主存的状态后端。它还可以利用本地磁盘进行缓存和缓冲。
+异步I/O模型用于读取和写入状态。详细信息请参阅[ForSt State Backend]({{< ref "docs/ops/state/state_backends#the-forststatebackend" >}}).
+* **新的状态 API**: 引入了新的状态 API (State V2) 用于异步状态读取和写入,异步状态访问是存算分离克服高网络延迟的必要组件。
+详细信息请参阅[New State APIs]({{< ref "docs/dev/datastream/fault-tolerance/state_v2" >}}).
+* **SQL 支持**: 许多 SQL 算子都被重写以支持存算分离和异步状态访问。用户可以轻松地通过配置项启用这些功能。
{{< hint info >}}
-Disaggregated state and asynchronous state access are encouraged for large state. However, when
-the state size is small, the local state management with synchronous state access is a better
-choice.
+对大状态作业,推荐使用存算分离和异步状态访问。如果状态很小,本地状态管理和同步状态访问是更好的选择。
Review Comment:
```suggestion
对于大状态作业,我们推荐使用存算分离和异步状态访问。如果作业状态很小,本地状态管理和同步状态访问会是更好的选择。
```
##########
docs/content.zh/docs/ops/state/state_backends.md:
##########
@@ -99,11 +99,35 @@ EmbeddedRocksDBStateBackend 是目前唯一支持增量 CheckPoint 的 State Bac
<a name="choose-the-right-state-backend"></a>
+### ForStStateBackend
+
+*ForStStateBackend* 是一个基于 [ForSt project](https://github.com/ververica/ForSt) 的状态后端。ForSt 也是一个 LSM-tree 结构的 key-value store,基于 RocksDB 构建。
+*ForStStateBackend* 为了而存算分离设计,更多细节请参考 [这里]({{< ref "docs/ops/state/disaggregated_state" >}})。
+最重要的是,它可以将 sst 文件保存在 Flink 支持的远程文件系统上,如 HDFS、S3 等。这允许 Flink 扩展状态大小,而不用受限于 TaskManager 的本地磁盘容量。
+同时,将 sst 文件保存在远程文件系统上,可以提供更轻量级的 checkpoint 和恢复方式。
+
+ForStStateBackend 还处于实验阶段,目前还不能用于生产环境。它只支持异步增量快照。
+
+以下场景推荐使用 ForStStateBackend:
+- 大状态、长窗口、大 key/value 状态的作业,本地磁盘空间不足以来存储状态。
+- 全高可用的设置。
Review Comment:
```suggestion
- 需要高可用的作业,定期使用检查点并在遇到报错时恢复。
```
##########
docs/content.zh/docs/ops/state/disaggregated_state.md:
##########
@@ -89,105 +74,82 @@ execution.checkpointing.dir: s3://your-bucket/flink-checkpoints
table.exec.mini-batch.enabled: false
table.optimizer.agg-phase-strategy: ONE_PHASE
```
-Thus, you could leverage the disaggregated state management and asynchronous state access in
-your SQL jobs. We haven't implemented the full support for the asynchronous state access
-in SQL yet. If the SQL operators you are using are not supported, the operator will fall back
-to the synchronous state implementation automatically. The performance may not be optimal in
-this case. The supported stateful operators are:
- - Rank (Top1, Append TopN)
- - Row Time Deduplicate
- - Aggregate (without distinct)
- - Join
- - Window Join
- - Tumble / Hop / Cumulative Window Aggregate
-
+如此一来,您可以在 SQL 作业中使用存算分离和异步状态访问。目前还未全部实现所有 SQL 算子的异步状态访问。
+对于还未支持异步状态访问的算子,算子会自动回退到同步状态实现,这种情况下性能可能不是最佳的。
+目前支持异步状态访问的有状态算子有:
Review Comment:
```suggestion
如此一来,您可以在 SQL 作业中使用存算分离和异步状态访问。目前我们并未实现所有异步状态访问版本的 SQL 算子。
对于还未支持异步状态访问的算子,算子会自动回退到同步状态实现,这种情况下性能可能不是最佳的。
目前支持异步状态访问的有状态算子有:
```
##########
docs/content.zh/docs/ops/state/state_backends.md:
##########
@@ -99,11 +99,35 @@ EmbeddedRocksDBStateBackend 是目前唯一支持增量 CheckPoint 的 State Bac
<a name="choose-the-right-state-backend"></a>
+### ForStStateBackend
+
+*ForStStateBackend* 是一个基于 [ForSt project](https://github.com/ververica/ForSt) 的状态后端。ForSt 也是一个 LSM-tree 结构的 key-value store,基于 RocksDB 构建。
+*ForStStateBackend* 为了而存算分离设计,更多细节请参考 [这里]({{< ref "docs/ops/state/disaggregated_state" >}})。
+最重要的是,它可以将 sst 文件保存在 Flink 支持的远程文件系统上,如 HDFS、S3 等。这允许 Flink 扩展状态大小,而不用受限于 TaskManager 的本地磁盘容量。
+同时,将 sst 文件保存在远程文件系统上,可以提供更轻量级的 checkpoint 和恢复方式。
+
+ForStStateBackend 还处于实验阶段,目前还不能用于生产环境。它只支持异步增量快照。
Review Comment:
```suggestion
ForStStateBackend 还处于实验阶段,目前不推荐用于生产环境。它当前只支持异步增量快照。
```
##########
docs/content.zh/docs/ops/state/disaggregated_state.md:
##########
@@ -89,105 +74,82 @@ execution.checkpointing.dir: s3://your-bucket/flink-checkpoints
table.exec.mini-batch.enabled: false
table.optimizer.agg-phase-strategy: ONE_PHASE
```
-Thus, you could leverage the disaggregated state management and asynchronous state access in
-your SQL jobs. We haven't implemented the full support for the asynchronous state access
-in SQL yet. If the SQL operators you are using are not supported, the operator will fall back
-to the synchronous state implementation automatically. The performance may not be optimal in
-this case. The supported stateful operators are:
- - Rank (Top1, Append TopN)
- - Row Time Deduplicate
- - Aggregate (without distinct)
- - Join
- - Window Join
- - Tumble / Hop / Cumulative Window Aggregate
-
+如此一来,您可以在 SQL 作业中使用存算分离和异步状态访问。目前还未全部实现所有 SQL 算子的异步状态访问。
+对于还未支持异步状态访问的算子,算子会自动回退到同步状态实现,这种情况下性能可能不是最佳的。
+目前支持异步状态访问的有状态算子有:
+- Rank (Top1, Append TopN)
+- Row Time Deduplicate
+- Aggregate (without distinct)
+- Join
+- Window Join
+- Tumble / Hop / Cumulative Window Aggregate
### For DataStream Jobs
-
-To enable the disaggregated state management in DataStream jobs, firstly you should use
-the `ForStStateBackend`. Configure via code in per-job mode:
+在 DataStream 作业启用存算分离,首先需要使用 `ForStStateBackend`. 通过以下代码在作业中配置:
```java
Configuration config = new Configuration();
config.set(StateBackendOptions.STATE_BACKEND, "forst");
config.set(CheckpointingOptions.CHECKPOINTS_DIRECTORY, "s3://your-bucket/flink-checkpoints");
config.set(CheckpointingOptions.INCREMENTAL_CHECKPOINTS, true);
env.configure(config);
```
-Or configure via `config.yaml`:
+或者通过 `config.yaml` 进行配置:
```yaml
state.backend.type: forst
# enable checkpoints, checkpoint directory is required
execution.checkpointing.incremental: true
execution.checkpointing.dir: s3://your-bucket/flink-checkpoints
```
+然后,您可以使用新的状态 API 来编写您的 datastream 作业。有关新的状态 API,请参阅[状态 V2]({{< ref "docs/dev/datastream/fault-tolerance/state_v2" >}}).
-Then, you should write your datastream jobs with the new state APIs. For more
-details, see [State V2]({{< ref "docs/dev/datastream/fault-tolerance/state_v2" >}}).
-
-## Advanced Tuning Options
-
-### Tuning ForSt State Backend
+## 高级调优选项
-The `ForStStateBackend` has many configurations to tune the performance.
-The design of ForSt is very similar to RocksDB, and the configurable options are almost the same,
-so you can refer to [large state tuning]({{< ref "docs/ops/state/large_state_tuning#tuning-rocksdb-or-forst" >}})
-to tune the ForSt state backend.
+### ForSt 状态后端调优
-Besides that, the following sections introduce some unique configurations for ForSt.
+`ForStStateBackend` 有许多配置项用于性能调优。ForSt 的设计和 RocksDB 非常类似,因此您可以参考[大状态调优]({{< ref "docs/ops/state/large_state_tuning#tuning-rocksdb-or-forst" >}})
+来为 ForSt state backend 调优。
+除此之外,以下章节介绍了一些 ForSt 独有的配置项。
-#### ForSt Primary Storage Location
+#### ForSt 主要存储位置
-By default, ForSt stores the state in the checkpoint directory. In this case,
-ForSt could perform lightweight checkpoints and fast recovery. However, users may
-want to store the state in a different location, e.g., a different bucket in S3.
-You can set the following configuration to specify the primary storage location:
+默认情况下,ForSt 会将状态存储在 checkpoint 目录中。在这种情况下,ForSt 可以执行轻量级的 checkpoint 和快速恢复。
+用户也可能想将状态存在一个不同的位置,例如,一个不同的 S3 桶。您可以通过以下配置来指定主要存储位置:
```yaml
state.backend.forst.primary-dir: s3://your-bucket/forst-state
```
+**注意**:如果设置了此配置,则您可能无法利用轻量级检查点和快速恢复,因为 ForSt 会在 checkpoint 和恢复过程中将文件
+从主要存储位置复制到 checkpoint 目录。
-**Note**: If you set this configuration, you may not be able to leverage the lightweight
-checkpoint and fast recovery, since the ForSt will perform file copy between the primary
-storage location and the checkpoint directory during checkpointing and recovery.
+#### ForSt 文件缓存
-
-#### ForSt File Cache
-
-ForSt uses the local disk for caching and buffering. The granularity of the cache is whole file.
-This is enabled by default, except when the primary storage location is set to local.
-There are two capacity limit policies for the cache:
- - Size-based: The cache will evict the oldest files when the cache size exceeds the limit.
- - Reserved-based: The cache will evict the oldest files when the reserved space on disk
-(the disk where cache directory is) is not enough.
-Corresponding configurations are:
+ForSt 使用本地磁盘进行缓存和缓冲。缓存粒度是整个文件。默认启用缓存功能,除非将主要存储位置设置为本地。缓存有两种容量限制策略:
+- Size-based: 在缓存大小超过限制时淘汰最旧的文件。
+- Reserved-based: 在磁盘上(缓存目录所在的磁盘)剩余空间不足淘汰最旧的文件。
Review Comment:
```suggestion
- 基于固定大小的 (Size-based): 在已使用缓存大小超过限制时淘汰最旧的文件。
- 基于预留空间的 (Reserved-based): 在磁盘上(缓存目录所在的磁盘)剩余空间不足时淘汰最旧的文件。
```
##########
docs/content.zh/docs/ops/state/disaggregated_state.md:
##########
@@ -25,58 +25,43 @@ specific language governing permissions and limitations
under the License.
-->
-# Disaggregated State Management
-
-## Overview
-
-For the first ten years of Flink, the state management is based on memory or local disk of the TaskManager.
-This approach works well for most use cases, but it has some limitations:
- * **Local Disk Constraints**: The state size is limited by the memory or disk size of the TaskManager.
- * **Spiky Resource Usage**: The local state model triggers periodic CPU and network I/O bursts during checkpointing or SST files compaction.
- * **Heavy Recovery**: State needs to be downloaded during recovery. The recovery time is
-proportional to the state size, which can be slow for large state sizes.
-
-In Flink 2.0, we introduced the disaggregated state management. This feature allows users to store
-the state in external storage systems like S3, HDFS, etc. This is useful when the state size
-is extremely large. It could be used to store the state in a more cost-effective way, or to
-persist or recovery the state in a more lightweight way. The benefits of disaggregated state management are:
- * **Unlimited State Size**: The state size is only limited by the external storage system.
- * **Stable Resource Usage**: The state is stored in external storage, thus the checkpoint could be very lightweight.
-And the SST files compaction could be done remotely (TODO).
- * **Fast Recovery**: No need to download the state during recovery. The recovery time is
-independent of the state size.
- * **Flexible**: Users can easily choose different external storage systems or I/O performance levels,
-or scale the storage based on their requirements without change their hardware.
- * **Cost-effective**: External storage are usually cheaper than local disk. Users can flexibly
-adjust computing resources and storage resources independently if there is any bottleneck.
-
-The disaggregated state management contains three parts:
- * **ForSt State Backend**: A state backend that stores the state in external storage systems. It
-can also leverage the local disk for caching and buffering. The asynchronous I/O model is used to
-read and write the state. For more details, see [ForSt State Backend]({{< ref "docs/ops/state/state_backends#the-forststatebackend" >}}).
- * **New State APIs**: The new state APIs (State V2) are introduced to perform asynchronous state
-reads and writes, which is essential for overcoming the high network latency when accessing
-the disaggregated state. For more details, see [New State APIs]({{< ref "docs/dev/datastream/fault-tolerance/state_v2" >}}).
- * **SQL Support**: Many SQL operators are rewritten to support the disaggregated state management
-and asynchronous state access. User can easily enable these by setting the configuration.
+# 存算分离(Disaggregated State Management)
+
+## 概览
+
+在 Flink 的前 10 年,状态管理是基于 TaskManager 的内存或本地磁盘。该方法适用于大多数用户场景,但存在一些限制:
+* **本地磁盘限制**: 状态大小受限于 TaskManager 的内存或者磁盘大小。
+* **资源使用尖刺**:本地状态模型会在 checkpoint 时触发 SST 文件的 compaction,引起 CPU 和网络 I/O 的尖刺。
+* **恢复繁重**:在恢复期间,需要下载状态。恢复时间与状态大小成比例,大状态的恢复非常慢。
+
+在 Flink 2.0,我们引入了存算分离(disaggregated state management). 存算分离允许用户将状态直接存储在第三方存储系统,如 S3,HDFS 等,
+这在状态非常大时很有用。存算分离以一种更有效的方式存储状态,轻量地持久化或恢复状态。存算分离有如下好处:
+* **状态大小无限制**:状态大小仅取决于外部存储系统大小。
+* **资源使用稳定**:状态被存储在外部存储上,因此 checkpoint 可以非常轻量。SST 文件的 compaction 也可以被放在远程(TODO)。
+* **恢复快速**:在恢复期间无需下载状态到本地,恢复时间与状态大小无关。
+* **灵活**:用户可以轻松地选择不同的外部存储系统或I/O性能级别,或者根据需求独立地扩展存储资源,而不需要改变他们的硬件。
+* **经济实惠**:外部存储通常比本地磁盘便宜。如果有瓶颈,用户可以灵活、独立地调整计算资源以及存储资源。
+
+存算分离包括三个部分:
+* **ForSt 状态后端**: 一个以外部存储系统作为主存的状态后端。它还可以利用本地磁盘进行缓存和缓冲。
+异步I/O模型用于读取和写入状态。详细信息请参阅[ForSt State Backend]({{< ref "docs/ops/state/state_backends#the-forststatebackend" >}}).
+* **新的状态 API**: 引入了新的状态 API (State V2) 用于异步状态读取和写入,异步状态访问是存算分离克服高网络延迟的必要组件。
+详细信息请参阅[New State APIs]({{< ref "docs/dev/datastream/fault-tolerance/state_v2" >}}).
+* **SQL 支持**: 许多 SQL 算子都被重写以支持存算分离和异步状态访问。用户可以轻松地通过配置项启用这些功能。
Review Comment:
```suggestion
存算分离包括三个部分:
* **ForSt 状态后端**: 一个以外部存储系统作为主存的状态后端,它可以利用本地磁盘进行缓存。
它使用异步 I/O 模型用于读取和写入状态。详细信息请参阅[ForSt State Backend]({{< ref "docs/ops/state/state_backends#the-forststatebackend" >}}).
* **新的状态 API**: 引入了新的状态 API (State V2) 用于异步状态读取和写入,异步状态访问是存算分离克服高网络延迟的必要组件。
详细信息请参阅[New State APIs]({{< ref "docs/dev/datastream/fault-tolerance/state_v2" >}}).
* **SQL 支持**: 许多 SQL 算子使用异步状态 API 来重新实现,用户可以轻松地通过配置项启用这些算子。
```
##########
docs/content.zh/docs/ops/state/state_backends.md:
##########
@@ -99,11 +99,35 @@ EmbeddedRocksDBStateBackend 是目前唯一支持增量 CheckPoint 的 State Bac
<a name="choose-the-right-state-backend"></a>
+### ForStStateBackend
+
+*ForStStateBackend* 是一个基于 [ForSt project](https://github.com/ververica/ForSt) 的状态后端。ForSt 也是一个 LSM-tree 结构的 key-value store,基于 RocksDB 构建。
+*ForStStateBackend* 为了而存算分离设计,更多细节请参考 [这里]({{< ref "docs/ops/state/disaggregated_state" >}})。
+最重要的是,它可以将 sst 文件保存在 Flink 支持的远程文件系统上,如 HDFS、S3 等。这允许 Flink 扩展状态大小,而不用受限于 TaskManager 的本地磁盘容量。
+同时,将 sst 文件保存在远程文件系统上,可以提供更轻量级的 checkpoint 和恢复方式。
+
+ForStStateBackend 还处于实验阶段,目前还不能用于生产环境。它只支持异步增量快照。
+
+以下场景推荐使用 ForStStateBackend:
+- 大状态、长窗口、大 key/value 状态的作业,本地磁盘空间不足以来存储状态。
+- 全高可用的设置。
+- 倾向异步状态访问。ForStStateBackend 是目前唯一支持异步状态访问的状态后端。
+- 需要轻量级的 checkpoint 和恢复的作业,例如云原生应用。
Review Comment:
```suggestion
- 需要轻量级的执行检查点和恢复的作业,例如云原生应用。
```
##########
docs/content.zh/docs/ops/state/disaggregated_state.md:
##########
@@ -25,58 +25,43 @@ specific language governing permissions and limitations
under the License.
-->
-# Disaggregated State Management
-
-## Overview
-
-For the first ten years of Flink, the state management is based on memory or local disk of the TaskManager.
-This approach works well for most use cases, but it has some limitations:
- * **Local Disk Constraints**: The state size is limited by the memory or disk size of the TaskManager.
- * **Spiky Resource Usage**: The local state model triggers periodic CPU and network I/O bursts during checkpointing or SST files compaction.
- * **Heavy Recovery**: State needs to be downloaded during recovery. The recovery time is
-proportional to the state size, which can be slow for large state sizes.
-
-In Flink 2.0, we introduced the disaggregated state management. This feature allows users to store
-the state in external storage systems like S3, HDFS, etc. This is useful when the state size
-is extremely large. It could be used to store the state in a more cost-effective way, or to
-persist or recovery the state in a more lightweight way. The benefits of disaggregated state management are:
- * **Unlimited State Size**: The state size is only limited by the external storage system.
- * **Stable Resource Usage**: The state is stored in external storage, thus the checkpoint could be very lightweight.
-And the SST files compaction could be done remotely (TODO).
- * **Fast Recovery**: No need to download the state during recovery. The recovery time is
-independent of the state size.
- * **Flexible**: Users can easily choose different external storage systems or I/O performance levels,
-or scale the storage based on their requirements without change their hardware.
- * **Cost-effective**: External storage are usually cheaper than local disk. Users can flexibly
-adjust computing resources and storage resources independently if there is any bottleneck.
-
-The disaggregated state management contains three parts:
- * **ForSt State Backend**: A state backend that stores the state in external storage systems. It
-can also leverage the local disk for caching and buffering. The asynchronous I/O model is used to
-read and write the state. For more details, see [ForSt State Backend]({{< ref "docs/ops/state/state_backends#the-forststatebackend" >}}).
- * **New State APIs**: The new state APIs (State V2) are introduced to perform asynchronous state
-reads and writes, which is essential for overcoming the high network latency when accessing
-the disaggregated state. For more details, see [New State APIs]({{< ref "docs/dev/datastream/fault-tolerance/state_v2" >}}).
- * **SQL Support**: Many SQL operators are rewritten to support the disaggregated state management
-and asynchronous state access. User can easily enable these by setting the configuration.
+# 存算分离(Disaggregated State Management)
+
+## 概览
+
+在 Flink 的前 10 年,状态管理是基于 TaskManager 的内存或本地磁盘。该方法适用于大多数用户场景,但存在一些限制:
+* **本地磁盘限制**: 状态大小受限于 TaskManager 的内存或者磁盘大小。
+* **资源使用尖刺**:本地状态模型会在 checkpoint 时触发 SST 文件的 compaction,引起 CPU 和网络 I/O 的尖刺。
+* **恢复繁重**:在恢复期间,需要下载状态。恢复时间与状态大小成比例,大状态的恢复非常慢。
+
+在 Flink 2.0,我们引入了存算分离(disaggregated state management). 存算分离允许用户将状态直接存储在第三方存储系统,如 S3,HDFS 等,
+这在状态非常大时很有用。存算分离以一种更有效的方式存储状态,轻量地持久化或恢复状态。存算分离有如下好处:
+* **状态大小无限制**:状态大小仅取决于外部存储系统大小。
+* **资源使用稳定**:状态被存储在外部存储上,因此 checkpoint 可以非常轻量。SST 文件的 compaction 也可以被放在远程(TODO)。
+* **恢复快速**:在恢复期间无需下载状态到本地,恢复时间与状态大小无关。
+* **灵活**:用户可以轻松地选择不同的外部存储系统或I/O性能级别,或者根据需求独立地扩展存储资源,而不需要改变他们的硬件。
+* **经济实惠**:外部存储通常比本地磁盘便宜。如果有瓶颈,用户可以灵活、独立地调整计算资源以及存储资源。
Review Comment:
```suggestion
在 Flink 2.0,我们引入了存算分离(disaggregated state management)来解决上述问题。 存算分离允许状态后端将状态直接存储在远端存储系统,如 S3,HDFS 等。
存算分离以一种更有效的方式存储状态,能够轻量地持久化或恢复状态。它有如下好处:
* **状态大小无限制**:状态大小仅取决于外部存储系统大小。
* **资源使用稳定**:状态被存储在外部存储上,因此 checkpoint 可以非常轻量。SST 文件的 compaction 也可以被放在远程(TODO)。
* **恢复快速**:在恢复期间无需下载状态到本地,恢复时间与状态大小无关。
* **灵活**:用户可以轻松地选择不同的外部存储系统或I/O性能级别,或者根据需求独立地扩展存储资源,而不需要改变他们的硬件。
* **经济实惠**:外部存储通常比本地磁盘便宜。如果有瓶颈,用户可以灵活、独立地调整计算资源以及存储资源。
```
##########
docs/content.zh/docs/ops/state/disaggregated_state.md:
##########
@@ -89,105 +74,82 @@ execution.checkpointing.dir: s3://your-bucket/flink-checkpoints
table.exec.mini-batch.enabled: false
table.optimizer.agg-phase-strategy: ONE_PHASE
```
-Thus, you could leverage the disaggregated state management and asynchronous state access in
-your SQL jobs. We haven't implemented the full support for the asynchronous state access
-in SQL yet. If the SQL operators you are using are not supported, the operator will fall back
-to the synchronous state implementation automatically. The performance may not be optimal in
-this case. The supported stateful operators are:
- - Rank (Top1, Append TopN)
- - Row Time Deduplicate
- - Aggregate (without distinct)
- - Join
- - Window Join
- - Tumble / Hop / Cumulative Window Aggregate
-
+如此一来,您可以在 SQL 作业中使用存算分离和异步状态访问。目前还未全部实现所有 SQL 算子的异步状态访问。
+对于还未支持异步状态访问的算子,算子会自动回退到同步状态实现,这种情况下性能可能不是最佳的。
+目前支持异步状态访问的有状态算子有:
+- Rank (Top1, Append TopN)
+- Row Time Deduplicate
+- Aggregate (without distinct)
+- Join
+- Window Join
+- Tumble / Hop / Cumulative Window Aggregate
### For DataStream Jobs
-
-To enable the disaggregated state management in DataStream jobs, firstly you should use
-the `ForStStateBackend`. Configure via code in per-job mode:
+在 DataStream 作业启用存算分离,首先需要使用 `ForStStateBackend`. 通过以下代码在作业中配置:
```java
Configuration config = new Configuration();
config.set(StateBackendOptions.STATE_BACKEND, "forst");
config.set(CheckpointingOptions.CHECKPOINTS_DIRECTORY, "s3://your-bucket/flink-checkpoints");
config.set(CheckpointingOptions.INCREMENTAL_CHECKPOINTS, true);
env.configure(config);
```
-Or configure via `config.yaml`:
+或者通过 `config.yaml` 进行配置:
```yaml
state.backend.type: forst
# enable checkpoints, checkpoint directory is required
execution.checkpointing.incremental: true
execution.checkpointing.dir: s3://your-bucket/flink-checkpoints
```
+然后,您可以使用新的状态 API 来编写您的 datastream 作业。有关新的状态 API,请参阅[状态 V2]({{< ref "docs/dev/datastream/fault-tolerance/state_v2" >}}).
-Then, you should write your datastream jobs with the new state APIs. For more
-details, see [State V2]({{< ref "docs/dev/datastream/fault-tolerance/state_v2" >}}).
-
-## Advanced Tuning Options
-
-### Tuning ForSt State Backend
+## 高级调优选项
-The `ForStStateBackend` has many configurations to tune the performance.
-The design of ForSt is very similar to RocksDB, and the configurable options are almost the same,
-so you can refer to [large state tuning]({{< ref "docs/ops/state/large_state_tuning#tuning-rocksdb-or-forst" >}})
-to tune the ForSt state backend.
+### ForSt 状态后端调优
-Besides that, the following sections introduce some unique configurations for ForSt.
+`ForStStateBackend` 有许多配置项用于性能调优。ForSt 的设计和 RocksDB 非常类似,因此您可以参考[大状态调优]({{< ref "docs/ops/state/large_state_tuning#tuning-rocksdb-or-forst" >}})
+来为 ForSt state backend 调优。
+除此之外,以下章节介绍了一些 ForSt 独有的配置项。
-#### ForSt Primary Storage Location
+#### ForSt 主要存储位置
-By default, ForSt stores the state in the checkpoint directory. In this case,
-ForSt could perform lightweight checkpoints and fast recovery. However, users may
-want to store the state in a different location, e.g., a different bucket in S3.
-You can set the following configuration to specify the primary storage location:
+默认情况下,ForSt 会将状态存储在 checkpoint 目录中。在这种情况下,ForSt 可以执行轻量级的 checkpoint 和快速恢复。
+用户也可能想将状态存在一个不同的位置,例如,一个不同的 S3 桶。您可以通过以下配置来指定主要存储位置:
```yaml
state.backend.forst.primary-dir: s3://your-bucket/forst-state
```
+**注意**:如果设置了此配置,则您可能无法利用轻量级检查点和快速恢复,因为 ForSt 会在 checkpoint 和恢复过程中将文件
+从主要存储位置复制到 checkpoint 目录。
-**Note**: If you set this configuration, you may not be able to leverage the lightweight
-checkpoint and fast recovery, since the ForSt will perform file copy between the primary
-storage location and the checkpoint directory during checkpointing and recovery.
+#### ForSt 文件缓存
-
-#### ForSt File Cache
-
-ForSt uses the local disk for caching and buffering. The granularity of the cache is whole file.
-This is enabled by default, except when the primary storage location is set to local.
-There are two capacity limit policies for the cache:
- - Size-based: The cache will evict the oldest files when the cache size exceeds the limit.
- - Reserved-based: The cache will evict the oldest files when the reserved space on disk
-(the disk where cache directory is) is not enough.
-Corresponding configurations are:
+ForSt 使用本地磁盘进行缓存和缓冲。缓存粒度是整个文件。默认启用缓存功能,除非将主要存储位置设置为本地。缓存有两种容量限制策略:
+- Size-based: 在缓存大小超过限制时淘汰最旧的文件。
+- Reserved-based: 在磁盘上(缓存目录所在的磁盘)剩余空间不足淘汰最旧的文件。
+相关配置项如下:
```yaml
state.backend.forst.cache.size-based-limit: 1GB
state.backend.forst.cache.reserve-size: 10GB
```
-Those can take effect together. If so, the cache will evict the oldest files when the cache
-size exceeds either the size-based limit or the reserved size limit.
+以上两个配置项可以同时生效。如果同时开启两个选项,在缓存大小超过 size-based 限制或者 reserved-based 限制之一时,缓存会淘汰最老的文件。
-One can also specify the cache directory via:
+可以通过以下配置项来指定缓存目录:
Review Comment:
```suggestion
您可以通过以下配置项来指定缓存目录:
```
##########
docs/content.zh/docs/ops/state/state_backends.md:
##########
@@ -99,11 +99,35 @@ EmbeddedRocksDBStateBackend 是目前唯一支持增量 CheckPoint 的 State Bac
<a name="choose-the-right-state-backend"></a>
+### ForStStateBackend
+
+*ForStStateBackend* 是一个基于 [ForSt project](https://github.com/ververica/ForSt) 的状态后端。ForSt 也是一个 LSM-tree 结构的 key-value store,基于 RocksDB 构建。
+*ForStStateBackend* 为了而存算分离设计,更多细节请参考 [这里]({{< ref "docs/ops/state/disaggregated_state" >}})。
+最重要的是,它可以将 sst 文件保存在 Flink 支持的远程文件系统上,如 HDFS、S3 等。这允许 Flink 扩展状态大小,而不用受限于 TaskManager 的本地磁盘容量。
+同时,将 sst 文件保存在远程文件系统上,可以提供更轻量级的 checkpoint 和恢复方式。
+
+ForStStateBackend 还处于实验阶段,目前还不能用于生产环境。它只支持异步增量快照。
+
+以下场景推荐使用 ForStStateBackend:
+- 大状态、长窗口、大 key/value 状态的作业,本地磁盘空间不足以来存储状态。
+- 全高可用的设置。
+- 倾向异步状态访问。ForStStateBackend 是目前唯一支持异步状态访问的状态后端。
+- 需要轻量级的 checkpoint 和恢复的作业,例如云原生应用。
+
+目前 ForStStateBackend 的限制:
+- 和 EmbeddedRocksDBStateBackend 一样,每个 key 和 value 的最大支持大小为 2^31 字节。
+- 不支持 canonical savepoint、全量快照、changelog 和小文件合并快照。只支持增量快照。
+
+相比 EmbeddedRocksDBStateBackend,ForStStateBackend 将数据保存在远程文件系统上,因此可以存储无限多的状态。TaskManager
+上的本地磁盘仅用来缓存文件,以提供更好的性能。请注意,当大部分的活跃状态都保存在远程文件系统上时,状态访问的性能可能会受网络延迟的影响。
+Flink 介绍了异步状态访问以缓解这个问题,您可以使用 State API V2 的异步状态访问方法来从异步状态访问中受益。请参考
+[State API V2 文档]({{< ref "docs/dev/datastream/fault-tolerance/state_v2" >}}) 以熟悉 State API V2。
+
## 选择合适的 State Backend
在选择 `HashMapStateBackend` 和 `RocksDB` 的时候,其实就是在性能与可扩展性之间权衡。`HashMapStateBackend` 是非常快的,因为每个状态的读取和算子对于 objects 的更新都是在 Java 的 heap 上;但是状态的大小受限于集群中可用的内存。
-另一方面,`RocksDB` 可以根据可用的 disk 空间扩展,并且只有它支持增量 snapshot。
-然而,每个状态的读取和更新都需要(反)序列化,而且在 disk 上进行读操作的性能可能要比基于内存的 state backend 慢一个数量级。
+另一方面,`RocksDB` 可以根据可用的 disk 空间扩展。 然而,每个状态的读取和更新都需要(反)序列化,而且在 disk 上进行读操作的性能可能要比基于内存的 state backend 慢一个数量级。
+如果您需要处理非常大的状态,并且状态大小超过可用的磁盘空间,或者更倾向于在云原生的设置下快速扩展,那么 `ForStStateBackend` 是一个不错的选择。
Review Comment:
```suggestion
另一方面,`RocksDB` 可以根据可用的本地磁盘空间扩展。 然而,每个状态的读取和更新都需要(反)序列化,而且在磁盘上进行读操作的性能可能要比基于内存的状态后端慢一个数量级。
如果您需要处理非常大的状态,并且状态大小超过可用的磁盘空间,或者更倾向于在云原生的设置下快速扩展,那么 `ForStStateBackend` 是一个不错的选择。
```
##########
docs/content.zh/docs/ops/state/state_backends.md:
##########
@@ -99,11 +99,35 @@ EmbeddedRocksDBStateBackend 是目前唯一支持增量 CheckPoint 的 State Bac
<a name="choose-the-right-state-backend"></a>
+### ForStStateBackend
+
+*ForStStateBackend* 是一个基于 [ForSt project](https://github.com/ververica/ForSt) 的状态后端。ForSt 也是一个 LSM-tree 结构的 key-value store,基于 RocksDB 构建。
+*ForStStateBackend* 为了而存算分离设计,更多细节请参考 [这里]({{< ref "docs/ops/state/disaggregated_state" >}})。
+最重要的是,它可以将 sst 文件保存在 Flink 支持的远程文件系统上,如 HDFS、S3 等。这允许 Flink 扩展状态大小,而不用受限于 TaskManager 的本地磁盘容量。
+同时,将 sst 文件保存在远程文件系统上,可以提供更轻量级的 checkpoint 和恢复方式。
+
+ForStStateBackend 还处于实验阶段,目前还不能用于生产环境。它只支持异步增量快照。
+
+以下场景推荐使用 ForStStateBackend:
+- 大状态、长窗口、大 key/value 状态的作业,本地磁盘空间不足以来存储状态。
+- 全高可用的设置。
+- 倾向异步状态访问。ForStStateBackend 是目前唯一支持异步状态访问的状态后端。
Review Comment:
```suggestion
- 希望使用异步状态访问。ForStStateBackend 是目前唯一支持异步状态访问的状态后端。
```
##########
docs/content.zh/docs/ops/state/state_backends.md:
##########
@@ -99,11 +99,35 @@ EmbeddedRocksDBStateBackend 是目前唯一支持增量 CheckPoint 的 State Bac
<a name="choose-the-right-state-backend"></a>
+### ForStStateBackend
+
+*ForStStateBackend* 是一个基于 [ForSt project](https://github.com/ververica/ForSt) 的状态后端。ForSt 也是一个 LSM-tree 结构的 key-value store,基于 RocksDB 构建。
+*ForStStateBackend* 为了而存算分离设计,更多细节请参考 [这里]({{< ref "docs/ops/state/disaggregated_state" >}})。
+最重要的是,它可以将 sst 文件保存在 Flink 支持的远程文件系统上,如 HDFS、S3 等。这允许 Flink 扩展状态大小,而不用受限于 TaskManager 的本地磁盘容量。
+同时,将 sst 文件保存在远程文件系统上,可以提供更轻量级的 checkpoint 和恢复方式。
+
+ForStStateBackend 还处于实验阶段,目前还不能用于生产环境。它只支持异步增量快照。
+
+以下场景推荐使用 ForStStateBackend:
+- 大状态、长窗口、大 key/value 状态的作业,本地磁盘空间不足以来存储状态。
+- 全高可用的设置。
+- 倾向异步状态访问。ForStStateBackend 是目前唯一支持异步状态访问的状态后端。
+- 需要轻量级的 checkpoint 和恢复的作业,例如云原生应用。
+
+目前 ForStStateBackend 的限制:
+- 和 EmbeddedRocksDBStateBackend 一样,每个 key 和 value 的最大支持大小为 2^31 字节。
+- 不支持 canonical savepoint、全量快照、changelog 和小文件合并快照。只支持增量快照。
+
+相比 EmbeddedRocksDBStateBackend,ForStStateBackend 将数据保存在远程文件系统上,因此可以存储无限多的状态。TaskManager
+上的本地磁盘仅用来缓存文件,以提供更好的性能。请注意,当大部分的活跃状态都保存在远程文件系统上时,状态访问的性能可能会受网络延迟的影响。
+Flink 介绍了异步状态访问以缓解这个问题,您可以使用 State API V2 的异步状态访问方法来从异步状态访问中受益。请参考
+[State API V2 文档]({{< ref "docs/dev/datastream/fault-tolerance/state_v2" >}}) 以熟悉 State API V2。
Review Comment:
```suggestion
相比 EmbeddedRocksDBStateBackend,ForStStateBackend 将数据保存在远程文件系统上,因此可以存储无限多的状态。TaskManager
上的本地磁盘仅用来缓存文件,以提供更好的性能。请注意,当大部分的活跃状态都保存在远程文件系统上时,状态访问的性能可能会受网络延迟的影响。
Flink 引入了异步状态访问以缓解这个问题,您可以使用 State API V2 的异步状态访问方法。请参考
[State API V2 文档]({{< ref "docs/dev/datastream/fault-tolerance/state_v2" >}}) 以熟悉并使用 State API V2。
```
##########
docs/content.zh/docs/ops/state/disaggregated_state.md:
##########
@@ -89,105 +74,82 @@ execution.checkpointing.dir: s3://your-bucket/flink-checkpoints
table.exec.mini-batch.enabled: false
table.optimizer.agg-phase-strategy: ONE_PHASE
```
-Thus, you could leverage the disaggregated state management and asynchronous state access in
-your SQL jobs. We haven't implemented the full support for the asynchronous state access
-in SQL yet. If the SQL operators you are using are not supported, the operator will fall back
-to the synchronous state implementation automatically. The performance may not be optimal in
-this case. The supported stateful operators are:
- - Rank (Top1, Append TopN)
- - Row Time Deduplicate
- - Aggregate (without distinct)
- - Join
- - Window Join
- - Tumble / Hop / Cumulative Window Aggregate
-
+如此一来,您可以在 SQL 作业中使用存算分离和异步状态访问。目前还未全部实现所有 SQL 算子的异步状态访问。
+对于还未支持异步状态访问的算子,算子会自动回退到同步状态实现,这种情况下性能可能不是最佳的。
+目前支持异步状态访问的有状态算子有:
+- Rank (Top1, Append TopN)
+- Row Time Deduplicate
+- Aggregate (without distinct)
+- Join
+- Window Join
+- Tumble / Hop / Cumulative Window Aggregate
### For DataStream Jobs
-
-To enable the disaggregated state management in DataStream jobs, firstly you should use
-the `ForStStateBackend`. Configure via code in per-job mode:
+在 DataStream 作业启用存算分离,首先需要使用 `ForStStateBackend`. 通过以下代码在作业中配置:
```java
Configuration config = new Configuration();
config.set(StateBackendOptions.STATE_BACKEND, "forst");
config.set(CheckpointingOptions.CHECKPOINTS_DIRECTORY, "s3://your-bucket/flink-checkpoints");
config.set(CheckpointingOptions.INCREMENTAL_CHECKPOINTS, true);
env.configure(config);
```
-Or configure via `config.yaml`:
+或者通过 `config.yaml` 进行配置:
```yaml
state.backend.type: forst
# enable checkpoints, checkpoint directory is required
execution.checkpointing.incremental: true
execution.checkpointing.dir: s3://your-bucket/flink-checkpoints
```
+然后,您可以使用新的状态 API 来编写您的 datastream 作业。有关新的状态 API,请参阅[状态 V2]({{< ref "docs/dev/datastream/fault-tolerance/state_v2" >}}).
-Then, you should write your datastream jobs with the new state APIs. For more
-details, see [State V2]({{< ref "docs/dev/datastream/fault-tolerance/state_v2" >}}).
-
-## Advanced Tuning Options
-
-### Tuning ForSt State Backend
+## 高级调优选项
-The `ForStStateBackend` has many configurations to tune the performance.
-The design of ForSt is very similar to RocksDB, and the configurable options are almost the same,
-so you can refer to [large state tuning]({{< ref "docs/ops/state/large_state_tuning#tuning-rocksdb-or-forst" >}})
-to tune the ForSt state backend.
+### ForSt 状态后端调优
-Besides that, the following sections introduce some unique configurations for ForSt.
+`ForStStateBackend` 有许多配置项用于性能调优。ForSt 的设计和 RocksDB 非常类似,因此您可以参考[大状态调优]({{< ref "docs/ops/state/large_state_tuning#tuning-rocksdb-or-forst" >}})
+来为 ForSt state backend 调优。
+除此之外,以下章节介绍了一些 ForSt 独有的配置项。
-#### ForSt Primary Storage Location
+#### ForSt 主要存储位置
-By default, ForSt stores the state in the checkpoint directory. In this case,
-ForSt could perform lightweight checkpoints and fast recovery. However, users may
-want to store the state in a different location, e.g., a different bucket in S3.
-You can set the following configuration to specify the primary storage location:
+默认情况下,ForSt 会将状态存储在 checkpoint 目录中。在这种情况下,ForSt 可以执行轻量级的 checkpoint 和快速恢复。
+用户也可能想将状态存在一个不同的位置,例如,一个不同的 S3 桶。您可以通过以下配置来指定主要存储位置:
```yaml
state.backend.forst.primary-dir: s3://your-bucket/forst-state
```
+**注意**:如果设置了此配置,则您可能无法利用轻量级检查点和快速恢复,因为 ForSt 会在 checkpoint 和恢复过程中将文件
+从主要存储位置复制到 checkpoint 目录。
-**Note**: If you set this configuration, you may not be able to leverage the lightweight
-checkpoint and fast recovery, since the ForSt will perform file copy between the primary
-storage location and the checkpoint directory during checkpointing and recovery.
+#### ForSt 文件缓存
-
-#### ForSt File Cache
-
-ForSt uses the local disk for caching and buffering. The granularity of the cache is whole file.
-This is enabled by default, except when the primary storage location is set to local.
-There are two capacity limit policies for the cache:
- - Size-based: The cache will evict the oldest files when the cache size exceeds the limit.
- - Reserved-based: The cache will evict the oldest files when the reserved space on disk
-(the disk where cache directory is) is not enough.
-Corresponding configurations are:
+ForSt 使用本地磁盘进行缓存和缓冲。缓存粒度是整个文件。默认启用缓存功能,除非将主要存储位置设置为本地。缓存有两种容量限制策略:
+- Size-based: 在缓存大小超过限制时淘汰最旧的文件。
+- Reserved-based: 在磁盘上(缓存目录所在的磁盘)剩余空间不足淘汰最旧的文件。
+相关配置项如下:
```yaml
state.backend.forst.cache.size-based-limit: 1GB
state.backend.forst.cache.reserve-size: 10GB
```
-Those can take effect together. If so, the cache will evict the oldest files when the cache
-size exceeds either the size-based limit or the reserved size limit.
+以上两个配置项可以同时生效。如果同时开启两个选项,在缓存大小超过 size-based 限制或者 reserved-based 限制之一时,缓存会淘汰最老的文件。
Review Comment:
```suggestion
以上两个配置项可以同时生效。如果同时开启两个选项,在缓存使用量触发以上两个限制之一时,缓存会淘汰最老的文件。
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37502][docs] Translate documentation for Disaggregated State Management into Chinese [flink]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #26312: [FLINK-37502][docs] Translate documentation for Disaggregated State Management into Chinese
URL: https://github.com/apache/flink/pull/26312
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org