You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Boris Lublinsky <bo...@lightbend.com> on 2019/05/07 19:55:54 UTC
Checkpointing and save pointing
I am planning to use external volume for this. My understanding is that it needs to be mounted only to the job manager, not the task managers. Is this correct, or it needs to be mounted to both?
Boris Lublinsky
FDP Architect
boris.lublinsky@lightbend.com
https://www.lightbend.com/
Re: Checkpointing and save pointing
Posted by Fabian Hueske <fh...@gmail.com>.
For checkpoints and savepoints, the JM and all TMs need access to the same
storage system.
This can be shared NFS that is mounted on each machine.
Best, Fabian
Am Fr., 10. Mai 2019 um 15:15 Uhr schrieb Boris Lublinsky <
boris.lublinsky@lightbend.com>:
> For now is a regular Link cluster,
> But even there I want to use both check and save pointing.
> We do not want to use Hadoop, but rather shared fs - NFS/Gluster.
> I was trying to see whether volumes need to be mounted only for Job
> manager or both.
> HA is the next step. Trying to find the code saving checkpointing location.
>
>
> Boris Lublinsky
> FDP Architect
> boris.lublinsky@lightbend.com
> https://www.lightbend.com/
>
> On May 10, 2019, at 2:47 AM, Fabian Hueske <fh...@gmail.com> wrote:
>
> Hi Boris,
>
> Is your question is in the context of replacing Zookeeper by a different
> service for highly-available setups or are you setting up a regular Flink
> cluster?
>
> Best, Fabian
>
>
>
> Am Mi., 8. Mai 2019 um 06:20 Uhr schrieb Congxian Qiu <
> qcx978132955@gmail.com>:
>
>> Hi, Boris
>>
>> TM will also need to write to the external volume.
>>
>> Best, Congxian
>> On May 8, 2019, 03:56 +0800, Boris Lublinsky <
>> boris.lublinsky@lightbend.com>, wrote:
>>
>> I am planning to use external volume for this. My understanding is that
>> it needs to be mounted only to the job manager, not the task managers. Is
>> this correct, or it needs to be mounted to both?
>>
>> Boris Lublinsky
>> FDP Architect
>> boris.lublinsky@lightbend.com
>> https://www.lightbend.com/
>>
>>
>
Re: Checkpointing and save pointing
Posted by Boris Lublinsky <bo...@lightbend.com>.
For now is a regular Link cluster,
But even there I want to use both check and save pointing.
We do not want to use Hadoop, but rather shared fs - NFS/Gluster.
I was trying to see whether volumes need to be mounted only for Job manager or both.
HA is the next step. Trying to find the code saving checkpointing location.
Boris Lublinsky
FDP Architect
boris.lublinsky@lightbend.com
https://www.lightbend.com/
> On May 10, 2019, at 2:47 AM, Fabian Hueske <fh...@gmail.com> wrote:
>
> Hi Boris,
>
> Is your question is in the context of replacing Zookeeper by a different service for highly-available setups or are you setting up a regular Flink cluster?
>
> Best, Fabian
>
>
>
> Am Mi., 8. Mai 2019 um 06:20 Uhr schrieb Congxian Qiu <qcx978132955@gmail.com <ma...@gmail.com>>:
> Hi, Boris
>
> TM will also need to write to the external volume.
>
> Best, Congxian
> On May 8, 2019, 03:56 +0800, Boris Lublinsky <boris.lublinsky@lightbend.com <ma...@lightbend.com>>, wrote:
>> I am planning to use external volume for this. My understanding is that it needs to be mounted only to the job manager, not the task managers. Is this correct, or it needs to be mounted to both?
>>
>> Boris Lublinsky
>> FDP Architect
>> boris.lublinsky@lightbend.com <ma...@lightbend.com>
>> https://www.lightbend.com/ <https://www.lightbend.com/>
Re: Checkpointing and save pointing
Posted by Fabian Hueske <fh...@gmail.com>.
Hi Boris,
Is your question is in the context of replacing Zookeeper by a different
service for highly-available setups or are you setting up a regular Flink
cluster?
Best, Fabian
Am Mi., 8. Mai 2019 um 06:20 Uhr schrieb Congxian Qiu <
qcx978132955@gmail.com>:
> Hi, Boris
>
> TM will also need to write to the external volume.
>
> Best, Congxian
> On May 8, 2019, 03:56 +0800, Boris Lublinsky <
> boris.lublinsky@lightbend.com>, wrote:
>
> I am planning to use external volume for this. My understanding is that it
> needs to be mounted only to the job manager, not the task managers. Is this
> correct, or it needs to be mounted to both?
>
> Boris Lublinsky
> FDP Architect
> boris.lublinsky@lightbend.com
> https://www.lightbend.com/
>
>
Re: Checkpointing and save pointing
Posted by Congxian Qiu <qc...@gmail.com>.
Hi, Boris
TM will also need to write to the external volume.
Best, Congxian
On May 8, 2019, 03:56 +0800, Boris Lublinsky <bo...@lightbend.com>, wrote:
> I am planning to use external volume for this. My understanding is that it needs to be mounted only to the job manager, not the task managers. Is this correct, or it needs to be mounted to both?
>
> Boris Lublinsky
> FDP Architect
> boris.lublinsky@lightbend.com
> https://www.lightbend.com/
>