You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by mi...@ubs.com on 2019/08/19 15:28:28 UTC
Recovery from job manager crash using check points
Hi,
I can use check points to recover Flink states when a task manger crashes.
I can not use check points to recover Flink states when a job manger crashes.
Do I need to set up zookeepers to keep the states when a job manager crashes?
Regards
Min
Re: Recovery from job manager crash using check points
Posted by Zili Chen <wa...@gmail.com>.
Hi Min,
For your question, the answer is no.
In standalone case Flink uses an in memory checkpoint store which
is able to restore your savepoint configured in command-line and
recover states from it.
Besides, stop with savepoint and resume the job from savepoint
is the standard path to migrate jobs.
Best,
tison.
<mi...@ubs.com> 于2019年8月21日周三 下午9:46写道：
> Thanks for the helpful reply.
>
>
>
> One more question, does this zookeeper or HA requirement apply for a
> savepoint?
>
>
>
> Can I bounce a single jobmanager cluster and rerun my flink job from its
> previous states with a save point directory? e.g.
>
> ./bin/flink run myJob.jar -s savepointDirectory
>
>
>
> Regards,
>
>
>
> Min
>
>
>
> *From:* Zili Chen [mailto:wander4096@gmail.com]
> *Sent:* Dienstag, 20. August 2019 04:16
> *To:* Biao Liu
> *Cc:* Tan, Min; user
> *Subject:* [External] Re: Recovery from job manager crash using check
> points
>
>
>
> Hi Min,
>
>
>
> I guess you use standalone high-availability and when TM fails,
>
> JM can recovered the job from an in-memory checkpoint store.
>
>
>
> However, when JM fails, since you don't persist state on ha backend
>
> such as ZooKeeper, even JM relaunched by YARN RM superseded by a
>
> stand by, the new one knows nothing about the previous jobs.
>
>
>
> In short, you need to set up ZooKeepers as you yourself mentioned.
>
>
>
> Best,
>
> tison.
>
>
>
>
>
> Biao Liu <mm...@gmail.com> 于2019年8月19日周一 下午11:49写道：
>
> Hi Min,
>
>
>
> > Do I need to set up zookeepers to keep the states when a job manager
> crashes?
>
>
>
> I guess you need to set up the HA [1] properly. Besides that, I would
> suggest you should also check the state backend.
>
>
>
> 1.
> https://ci.apache.org/projects/flink/flink-docs-master/ops/jobmanager_high_availability.html
>
> 2.
> https://ci.apache.org/projects/flink/flink-docs-master/ops/state/state_backends.html
>
>
>
> Thanks,
>
> Biao /'bɪ.aʊ/
>
>
>
>
>
>
>
> On Mon, 19 Aug 2019 at 23:28, <mi...@ubs.com> wrote:
>
> Hi,
>
>
>
> I can use check points to recover Flink states when a task manger crashes.
>
>
>
> I can not use check points to recover Flink states when a job manger
> crashes.
>
>
>
> Do I need to set up zookeepers to keep the states when a job manager
> crashes?
>
>
>
> Regards
>
>
>
> Min
>
>
>
>
RE: Recovery from job manager crash using check
points
Posted by mi...@ubs.com.
Thanks for the helpful reply.
One more question, does this zookeeper or HA requirement apply for a savepoint?
Can I bounce a single jobmanager cluster and rerun my flink job from its previous states with a save point directory? e.g.
./bin/flink run myJob.jar -s savepointDirectory
Regards,
Min
From: Zili Chen [mailto:wander4096@gmail.com]
Sent: Dienstag, 20. August 2019 04:16
To: Biao Liu
Cc: Tan, Min; user
Subject: [External] Re: Recovery from job manager crash using check points
Hi Min,
I guess you use standalone high-availability and when TM fails,
JM can recovered the job from an in-memory checkpoint store.
However, when JM fails, since you don't persist state on ha backend
such as ZooKeeper, even JM relaunched by YARN RM superseded by a
stand by, the new one knows nothing about the previous jobs.
In short, you need to set up ZooKeepers as you yourself mentioned.
Best,
tison.
Biao Liu <mm...@gmail.com>> 于2019年8月19日周一 下午11:49写道：
Hi Min,
> Do I need to set up zookeepers to keep the states when a job manager crashes?
I guess you need to set up the HA [1] properly. Besides that, I would suggest you should also check the state backend.
1. https://ci.apache.org/projects/flink/flink-docs-master/ops/jobmanager_high_availability.html
2. https://ci.apache.org/projects/flink/flink-docs-master/ops/state/state_backends.html
Thanks,
Biao /'bɪ.aʊ/
On Mon, 19 Aug 2019 at 23:28, <mi...@ubs.com>> wrote:
Hi,
I can use check points to recover Flink states when a task manger crashes.
I can not use check points to recover Flink states when a job manger crashes.
Do I need to set up zookeepers to keep the states when a job manager crashes?
Regards
Min
Re: Recovery from job manager crash using check points
Posted by Zili Chen <wa...@gmail.com>.
Hi Min,
I guess you use standalone high-availability and when TM fails,
JM can recovered the job from an in-memory checkpoint store.
However, when JM fails, since you don't persist state on ha backend
such as ZooKeeper, even JM relaunched by YARN RM superseded by a
stand by, the new one knows nothing about the previous jobs.
In short, you need to set up ZooKeepers as you yourself mentioned.
Best,
tison.
Biao Liu <mm...@gmail.com> 于2019年8月19日周一 下午11:49写道：
> Hi Min,
>
> > Do I need to set up zookeepers to keep the states when a job manager
> crashes?
>
> I guess you need to set up the HA [1] properly. Besides that, I would
> suggest you should also check the state backend.
>
> 1.
> https://ci.apache.org/projects/flink/flink-docs-master/ops/jobmanager_high_availability.html
> 2.
> https://ci.apache.org/projects/flink/flink-docs-master/ops/state/state_backends.html
>
> Thanks,
> Biao /'bɪ.aʊ/
>
>
>
> On Mon, 19 Aug 2019 at 23:28, <mi...@ubs.com> wrote:
>
>> Hi,
>>
>>
>>
>> I can use check points to recover Flink states when a task manger crashes.
>>
>>
>>
>> I can not use check points to recover Flink states when a job manger
>> crashes.
>>
>>
>>
>> Do I need to set up zookeepers to keep the states when a job manager
>> crashes?
>>
>>
>>
>> Regards
>>
>>
>>
>> Min
>>
>>
>>
>
Re: Recovery from job manager crash using check points
Posted by Biao Liu <mm...@gmail.com>.
Hi Min,
> Do I need to set up zookeepers to keep the states when a job manager
crashes?
I guess you need to set up the HA [1] properly. Besides that, I would
suggest you should also check the state backend.
1.
https://ci.apache.org/projects/flink/flink-docs-master/ops/jobmanager_high_availability.html
2.
https://ci.apache.org/projects/flink/flink-docs-master/ops/state/state_backends.html
Thanks,
Biao /'bɪ.aʊ/
On Mon, 19 Aug 2019 at 23:28, <mi...@ubs.com> wrote:
> Hi,
>
>
>
> I can use check points to recover Flink states when a task manger crashes.
>
>
>
> I can not use check points to recover Flink states when a job manger
> crashes.
>
>
>
> Do I need to set up zookeepers to keep the states when a job manager
> crashes?
>
>
>
> Regards
>
>
>
> Min
>
>
>
Re: Recovery from job manager crash using check points
Posted by miki haiat <mi...@gmail.com>.
Wich kind of deployment system are you using,
Standalone ,yarn ... Other ?
On Mon, Aug 19, 2019, 18:28 <mi...@ubs.com> wrote:
> Hi,
>
>
>
> I can use check points to recover Flink states when a task manger crashes.
>
>
>
> I can not use check points to recover Flink states when a job manger
> crashes.
>
>
>
> Do I need to set up zookeepers to keep the states when a job manager
> crashes?
>
>
>
> Regards
>
>
>
> Min
>
>
>