You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Tony Wei <to...@gmail.com> on 2018/08/17 02:38:17 UTC
Need a clarification about removing a stateful operator
Hi all,
I'm confused about the description in documentation. [1]
- *Removing a stateful operator:* The state of the removed operator is
lost unless
another operator takes it over. When starting the upgraded application,
you have
to explicitly agree to discard the state.
Does that mean if I take a full snapshot (e.g. savepoint) after restoring
by explicitly agreeing to
discard the state, then the state won't exist in that snapshot? Or does it
just mean ignore the
state but the state still exist forever, unless I explicitly purge that
state by using state operator?
And could this behavior differ between different state backend (Memory, FS,
RocksDB) ?
Many thanks,
Tony Wei
[1]
https://ci.apache.org/projects/flink/flink-docs-master/ops/upgrading.html#application-topology
Re: Need a clarification about removing a stateful operator
Posted by Tony Wei <to...@gmail.com>.
Hi Stefan,
Thanks for your detailed explanation.
Best,
Tony Wei
2018-08-17 15:56 GMT+08:00 Stefan Richter <s....@data-artisans.com>:
> Hi,
>
> it will not be transported. The JM does the state assignment to create the
> deployment information for all tasks. If will just exclude the state for
> operators that are not present. So in your next checkpoints they will no
> longer be contained.
>
> Best,
> Stefan
>
>
> Am 17.08.2018 um 09:26 schrieb Tony Wei <to...@gmail.com>:
>
> Hi Chesnay,
>
> Thanks for your quick reply. I have another question. Will the state,
> which is ignored, be transported
> to TMs from DFS? Or will it be detected by JM's checkpoint coordinator and
> only those states reuired
> by operators be transported to each TM?
>
> Best,
> Tony Wei
>
> 2018-08-17 14:38 GMT+08:00 Chesnay Schepler <ch...@apache.org>:
>
>> The state won't exist in the snapshot.
>>
>>
>> On 17.08.2018 04:38, Tony Wei wrote:
>>
>> Hi all,
>>
>> I'm confused about the description in documentation. [1]
>>
>>
>> - *Removing a stateful operator:* The state of the removed operator
>> is lost unless
>> another operator takes it over. When starting the upgraded
>> application, you have
>> to explicitly agree to discard the state.
>>
>> Does that mean if I take a full snapshot (e.g. savepoint) after restoring
>> by explicitly agreeing to
>> discard the state, then the state won't exist in that snapshot? Or does
>> it just mean ignore the
>> state but the state still exist forever, unless I explicitly purge that
>> state by using state operator?
>>
>> And could this behavior differ between different state backend (Memory,
>> FS, RocksDB) ?
>>
>> Many thanks,
>> Tony Wei
>>
>> [1] https://ci.apache.org/projects/flink/flink-docs-master/
>> ops/upgrading.html#application-topology
>>
>>
>>
>
>
Re: Need a clarification about removing a stateful operator
Posted by Stefan Richter <s....@data-artisans.com>.
Hi,
it will not be transported. The JM does the state assignment to create the deployment information for all tasks. If will just exclude the state for operators that are not present. So in your next checkpoints they will no longer be contained.
Best,
Stefan
> Am 17.08.2018 um 09:26 schrieb Tony Wei <to...@gmail.com>:
>
> Hi Chesnay,
>
> Thanks for your quick reply. I have another question. Will the state, which is ignored, be transported
> to TMs from DFS? Or will it be detected by JM's checkpoint coordinator and only those states reuired
> by operators be transported to each TM?
>
> Best,
> Tony Wei
>
> 2018-08-17 14:38 GMT+08:00 Chesnay Schepler <chesnay@apache.org <ma...@apache.org>>:
> The state won't exist in the snapshot.
>
>
> On 17.08.2018 04:38, Tony Wei wrote:
>> Hi all,
>>
>> I'm confused about the description in documentation. [1]
>>
>> Removing a stateful operator: The state of the removed operator is lost unless
>> another operator takes it over. When starting the upgraded application, you have
>> to explicitly agree to discard the state.
>> Does that mean if I take a full snapshot (e.g. savepoint) after restoring by explicitly agreeing to
>> discard the state, then the state won't exist in that snapshot? Or does it just mean ignore the
>> state but the state still exist forever, unless I explicitly purge that state by using state operator?
>>
>> And could this behavior differ between different state backend (Memory, FS, RocksDB) ?
>>
>> Many thanks,
>> Tony Wei
>>
>> [1] https://ci.apache.org/projects/flink/flink-docs-master/ops/upgrading.html#application-topology <https://ci.apache.org/projects/flink/flink-docs-master/ops/upgrading.html#application-topology>
>
Re: Need a clarification about removing a stateful operator
Posted by Tony Wei <to...@gmail.com>.
Hi Chesnay,
Thanks for your quick reply. I have another question. Will the state, which
is ignored, be transported
to TMs from DFS? Or will it be detected by JM's checkpoint coordinator and
only those states reuired
by operators be transported to each TM?
Best,
Tony Wei
2018-08-17 14:38 GMT+08:00 Chesnay Schepler <ch...@apache.org>:
> The state won't exist in the snapshot.
>
>
> On 17.08.2018 04:38, Tony Wei wrote:
>
> Hi all,
>
> I'm confused about the description in documentation. [1]
>
>
> - *Removing a stateful operator:* The state of the removed operator is
> lost unless
> another operator takes it over. When starting the upgraded
> application, you have
> to explicitly agree to discard the state.
>
> Does that mean if I take a full snapshot (e.g. savepoint) after restoring
> by explicitly agreeing to
> discard the state, then the state won't exist in that snapshot? Or does it
> just mean ignore the
> state but the state still exist forever, unless I explicitly purge that
> state by using state operator?
>
> And could this behavior differ between different state backend (Memory,
> FS, RocksDB) ?
>
> Many thanks,
> Tony Wei
>
> [1] https://ci.apache.org/projects/flink/flink-docs-
> master/ops/upgrading.html#application-topology
>
>
>
Re: Need a clarification about removing a stateful operator
Posted by Chesnay Schepler <ch...@apache.org>.
The state won't exist in the snapshot.
On 17.08.2018 04:38, Tony Wei wrote:
> Hi all,
>
> I'm confused about the description in documentation. [1]
>
> * *Removing a stateful operator:*The state of the removed operator
> is lost unless
> another operator takes it over. When starting the upgraded
> application, you have
> to explicitly agree to discard the state.
>
> Does that mean if I take a full snapshot (e.g. savepoint) after
> restoring by explicitly agreeing to
> discard the state, then the state won't exist in that snapshot? Or
> does it just mean ignore the
> state but the state still exist forever, unless I explicitly purge
> that state by using state operator?
>
> And could this behavior differ between different state backend
> (Memory, FS, RocksDB) ?
>
> Many thanks,
> Tony Wei
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-master/ops/upgrading.html#application-topology