You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Dongwoo Kim <do...@gmail.com> on 2023/04/09 14:23:01 UTC
Quick question about flink document.
Hi community, I’m new to flink and trying to learn about the concepts of flink to prepare migrating heron application to flink.
I have a quick question about this flink document.
(https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/concepts/stateful-stream-processing/#snapshotting-operator-state <https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/concepts/stateful-stream-processing/#snapshotting-operator-state>)
What I understood is states are stored in configured state backend which can be either task manager’s heap or rocksdb.
And snapshots of checkpoint is stored by default in job manager’s heap and mostly in distributed file system.
But in the document it says like below and it is confusing to me. Isn’t the second line talking about checkpoint storage or checkpoint backend? Not state backend? Thanks in advance, enjoy your weekend!
"Because the state of a snapshot may be large, it is stored in a configurable state backend <https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/state/state_backends/>. By default, this is the JobManager’s memory, but for production use a distributed reliable storage should be configured (such as HDFS)”
Re: Quick question about flink document.
Posted by Shammon FY <zj...@gmail.com>.
Hi Dongwoo
I think there are two configurations about state, one is state backend and
the other is snapshot storage. Flink will create a snapshot for each state
when the stateful operator collects all checkpoint barriers.
As @Feng mentioned above, users can config different state backend with
option: state.backend
The snapshot of state can be stored in JobManager. When the state is large,
flink supports storing the snapshot a distributed storage with option:
state.checkpoints.dir:
Best,
Shammon FY
On Mon, Apr 10, 2023 at 12:31 AM Feng Jin <ji...@gmail.com> wrote:
> Hi Dongwoo
>
>
> This can be quite confusing.
> Before Flink 1.13, Flink's statebackend was actually a hybrid concept that
> included three types of statebackends:
> *MemoryStateBackend*, *FsStateBackend*, and *RocksDBStateBackend*.
>
> The default *MemoryStateBackend* uses heap as the backend, and the state
> is stored in jobManger.
>
>
> You can refer to this migration document for more information:
> https://nightlies.apache.org/flink/flink-docs-master/docs/ops/state/state_backends/#migrating-from-legacy-backends
> .
>
>
> Best
> Feng
>
> On Sun, Apr 9, 2023 at 10:23 PM Dongwoo Kim <do...@gmail.com>
> wrote:
>
>> Hi community, I’m new to flink and trying to learn about the concepts of
>> flink to prepare migrating heron application to flink.
>> I have a quick question about this flink document.
>> (
>> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/concepts/stateful-stream-processing/#snapshotting-operator-state
>> )
>>
>> What I understood is states are stored in configured state backend which
>> can be either task manager’s heap or rocksdb.
>> And snapshots of checkpoint is stored by default in job manager’s heap
>> and mostly in distributed file system.
>> But in the document it says like below and it is confusing to me. Isn’t
>> the second line talking about checkpoint storage or checkpoint backend? Not
>> state backend? Thanks in advance, enjoy your weekend!
>>
>> *"Because the state of a snapshot may be large, it is stored in a
>> configurable state backend
>> <https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/state/state_backends/>.
>> By default, this is the JobManager’s memory, but for production use a
>> distributed reliable storage should be configured (such as HDFS)” *
>>
>
Re: Quick question about flink document.
Posted by Hector Rios <oj...@gmail.com>.
Hi Kim
I'm a little late to the party here, but firstly I have to agree with you
that the docs are confusing. Especially for newcomers to Flink, like
myself.
So, let me take a stab at it and try and explain how I understand the
paragraph you linked to above. First of all I think it's important to know
that "snapshots" in this context refers to checkpoints. You can infer this
by the use of the word "snapshot barriers", which is known in most cases as
checkpoint barriers.
Adding to the confusion is this sentence.
> *At that point, all updates to the state from records before the barriers
> have been made, and no updates that depend on records from after the
> barriers have been applied.*
As a native English speaker, I have no idea what this sentence even means.
So, let's just ignore this one for now. Let's get into the sentence you
highlighted.
>
> *Because the state of a snapshot may be large, it is stored in a
> configurable state backend
> <https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/state/state_backends/>.
> By default, this is the JobManager’s memory, but for production use a
> distributed reliable storage should be configured (such as HDFS). *
My understanding is that in this context "state backend" refers to
checkpoint storage and not where operator state is being while a job is
running, e.g, RocksDB. The reason why is the use of the phrase "distributed
reliable storage", which is not what something like RocksDB is.
Sorry for the long winded explanation, but to wrap up, I would agree with
you that this refers to checkpoint storage. Thank you for highlighting this
confusing piece of documentation. I thought I was the only one who was
confused. The syntax of talking "Flink" can be very confusing as some words
can be used in multiple contexts.
Best Regards
-Hector
On Fri, Apr 21, 2023 at 6:53 PM Dongwoo Kim <do...@gmail.com> wrote:
> Hi, Ruan
> Sorry for the late reply.
> As you mentioned above, I understood that this part describes the process
> of snaphoting the operator's state.
> And for the snapshots, they are stored to *checkpoint storage* which can
> be by default JobManager or other distributed file system such as hdfs.
>
> Thus, isn't the boxed part below more suitable for *checkpoint storage*
> rather than state backend?
> Since it is where the snapshot is stored and the explanation which follows
> right after mentions the jobmanager's memory or hdfs which is a candidate
> for checkpoint storage.
>
> [image: image.png]
> ref:
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/concepts/stateful-stream-processing/#snapshotting-operator-state
>
> Best,
> Kim
>
> 2023년 4월 10일 (월) 오전 11:25, Hang Ruan <ru...@gmail.com>님이 작성:
>
>> Hi, Dongwoo,
>>
>> I think there is no problem in this part. This part describes
>> snapshotting Operator State, which is checkpointing. The checkpoint will
>> store by the JobManager and use the checkpoint storage.
>>
>> Best,
>> Hang
>>
>> Feng Jin <ji...@gmail.com> 于2023年4月10日周一 00:32写道:
>>
>>> Hi Dongwoo
>>>
>>>
>>> This can be quite confusing.
>>> Before Flink 1.13, Flink's statebackend was actually a hybrid concept
>>> that included three types of statebackends:
>>> *MemoryStateBackend*, *FsStateBackend*, and *RocksDBStateBackend*.
>>>
>>> The default *MemoryStateBackend* uses heap as the backend, and the
>>> state is stored in jobManger.
>>>
>>>
>>> You can refer to this migration document for more information:
>>> https://nightlies.apache.org/flink/flink-docs-master/docs/ops/state/state_backends/#migrating-from-legacy-backends
>>> .
>>>
>>>
>>> Best
>>> Feng
>>>
>>> On Sun, Apr 9, 2023 at 10:23 PM Dongwoo Kim <do...@gmail.com>
>>> wrote:
>>>
>>>> Hi community, I’m new to flink and trying to learn about the concepts
>>>> of flink to prepare migrating heron application to flink.
>>>> I have a quick question about this flink document.
>>>> (
>>>> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/concepts/stateful-stream-processing/#snapshotting-operator-state
>>>> )
>>>>
>>>> What I understood is states are stored in configured state backend
>>>> which can be either task manager’s heap or rocksdb.
>>>> And snapshots of checkpoint is stored by default in job manager’s heap
>>>> and mostly in distributed file system.
>>>> But in the document it says like below and it is confusing to me. Isn’t
>>>> the second line talking about checkpoint storage or checkpoint backend? Not
>>>> state backend? Thanks in advance, enjoy your weekend!
>>>>
>>>> *"Because the state of a snapshot may be large, it is stored in a
>>>> configurable state backend
>>>> <https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/state/state_backends/>.
>>>> By default, this is the JobManager’s memory, but for production use a
>>>> distributed reliable storage should be configured (such as HDFS)” *
>>>>
>>>
Re: Quick question about flink document.
Posted by Dongwoo Kim <do...@gmail.com>.
Hi, Ruan
Sorry for the late reply.
As you mentioned above, I understood that this part describes the process
of snaphoting the operator's state.
And for the snapshots, they are stored to *checkpoint storage* which can be
by default JobManager or other distributed file system such as hdfs.
Thus, isn't the boxed part below more suitable for *checkpoint storage*
rather than state backend?
Since it is where the snapshot is stored and the explanation which follows
right after mentions the jobmanager's memory or hdfs which is a candidate
for checkpoint storage.
[image: image.png]
ref:
https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/concepts/stateful-stream-processing/#snapshotting-operator-state
Best,
Kim
2023년 4월 10일 (월) 오전 11:25, Hang Ruan <ru...@gmail.com>님이 작성:
> Hi, Dongwoo,
>
> I think there is no problem in this part. This part describes snapshotting
> Operator State, which is checkpointing. The checkpoint will store by the
> JobManager and use the checkpoint storage.
>
> Best,
> Hang
>
> Feng Jin <ji...@gmail.com> 于2023年4月10日周一 00:32写道:
>
>> Hi Dongwoo
>>
>>
>> This can be quite confusing.
>> Before Flink 1.13, Flink's statebackend was actually a hybrid concept
>> that included three types of statebackends:
>> *MemoryStateBackend*, *FsStateBackend*, and *RocksDBStateBackend*.
>>
>> The default *MemoryStateBackend* uses heap as the backend, and the state
>> is stored in jobManger.
>>
>>
>> You can refer to this migration document for more information:
>> https://nightlies.apache.org/flink/flink-docs-master/docs/ops/state/state_backends/#migrating-from-legacy-backends
>> .
>>
>>
>> Best
>> Feng
>>
>> On Sun, Apr 9, 2023 at 10:23 PM Dongwoo Kim <do...@gmail.com>
>> wrote:
>>
>>> Hi community, I’m new to flink and trying to learn about the concepts of
>>> flink to prepare migrating heron application to flink.
>>> I have a quick question about this flink document.
>>> (
>>> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/concepts/stateful-stream-processing/#snapshotting-operator-state
>>> )
>>>
>>> What I understood is states are stored in configured state backend which
>>> can be either task manager’s heap or rocksdb.
>>> And snapshots of checkpoint is stored by default in job manager’s heap
>>> and mostly in distributed file system.
>>> But in the document it says like below and it is confusing to me. Isn’t
>>> the second line talking about checkpoint storage or checkpoint backend? Not
>>> state backend? Thanks in advance, enjoy your weekend!
>>>
>>> *"Because the state of a snapshot may be large, it is stored in a
>>> configurable state backend
>>> <https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/state/state_backends/>.
>>> By default, this is the JobManager’s memory, but for production use a
>>> distributed reliable storage should be configured (such as HDFS)” *
>>>
>>
Re: Quick question about flink document.
Posted by Hang Ruan <ru...@gmail.com>.
Hi, Dongwoo,
I think there is no problem in this part. This part describes snapshotting
Operator State, which is checkpointing. The checkpoint will store by the
JobManager and use the checkpoint storage.
Best,
Hang
Feng Jin <ji...@gmail.com> 于2023年4月10日周一 00:32写道:
> Hi Dongwoo
>
>
> This can be quite confusing.
> Before Flink 1.13, Flink's statebackend was actually a hybrid concept that
> included three types of statebackends:
> *MemoryStateBackend*, *FsStateBackend*, and *RocksDBStateBackend*.
>
> The default *MemoryStateBackend* uses heap as the backend, and the state
> is stored in jobManger.
>
>
> You can refer to this migration document for more information:
> https://nightlies.apache.org/flink/flink-docs-master/docs/ops/state/state_backends/#migrating-from-legacy-backends
> .
>
>
> Best
> Feng
>
> On Sun, Apr 9, 2023 at 10:23 PM Dongwoo Kim <do...@gmail.com>
> wrote:
>
>> Hi community, I’m new to flink and trying to learn about the concepts of
>> flink to prepare migrating heron application to flink.
>> I have a quick question about this flink document.
>> (
>> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/concepts/stateful-stream-processing/#snapshotting-operator-state
>> )
>>
>> What I understood is states are stored in configured state backend which
>> can be either task manager’s heap or rocksdb.
>> And snapshots of checkpoint is stored by default in job manager’s heap
>> and mostly in distributed file system.
>> But in the document it says like below and it is confusing to me. Isn’t
>> the second line talking about checkpoint storage or checkpoint backend? Not
>> state backend? Thanks in advance, enjoy your weekend!
>>
>> *"Because the state of a snapshot may be large, it is stored in a
>> configurable state backend
>> <https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/state/state_backends/>.
>> By default, this is the JobManager’s memory, but for production use a
>> distributed reliable storage should be configured (such as HDFS)” *
>>
>
Re: Quick question about flink document.
Posted by Feng Jin <ji...@gmail.com>.
Hi Dongwoo
This can be quite confusing.
Before Flink 1.13, Flink's statebackend was actually a hybrid concept that
included three types of statebackends:
*MemoryStateBackend*, *FsStateBackend*, and *RocksDBStateBackend*.
The default *MemoryStateBackend* uses heap as the backend, and the state is
stored in jobManger.
You can refer to this migration document for more information:
https://nightlies.apache.org/flink/flink-docs-master/docs/ops/state/state_backends/#migrating-from-legacy-backends
.
Best
Feng
On Sun, Apr 9, 2023 at 10:23 PM Dongwoo Kim <do...@gmail.com> wrote:
> Hi community, I’m new to flink and trying to learn about the concepts of
> flink to prepare migrating heron application to flink.
> I have a quick question about this flink document.
> (
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/concepts/stateful-stream-processing/#snapshotting-operator-state
> )
>
> What I understood is states are stored in configured state backend which
> can be either task manager’s heap or rocksdb.
> And snapshots of checkpoint is stored by default in job manager’s heap and
> mostly in distributed file system.
> But in the document it says like below and it is confusing to me. Isn’t
> the second line talking about checkpoint storage or checkpoint backend? Not
> state backend? Thanks in advance, enjoy your weekend!
>
> *"Because the state of a snapshot may be large, it is stored in a
> configurable state backend
> <https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/state/state_backends/>.
> By default, this is the JobManager’s memory, but for production use a
> distributed reliable storage should be configured (such as HDFS)” *
>