You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Boris Lublinsky <bo...@lightbend.com> on 2019/05/24 21:22:47 UTC
Job cluster and HA
Hi,
I was experimenting with HA lately and see that it recovers successfully job, in the case of jobmanager restarts.
Now my question is whether it will work for the job cluster.
Based on the instructions https://github.com/apache/flink/blob/release-1.8/flink-container/docker/README.md <https://github.com/apache/flink/blob/release-1.8/flink-container/docker/README.md>
I can see https://github.com/apache/flink/blob/release-1.8/flink-container/docker/docker-entrypoint.sh <https://github.com/apache/flink/blob/release-1.8/flink-container/docker/docker-entrypoint.sh> that
In this case the following command is invoked:
exec $FLINK_HOME/bin/standalone-job.sh start-foreground "$@“
Which means that if a jobManager restarts, the following is going to happen:
1. It will use HA to restore job that was running
2. A new job will be submitted, overwriting restored job and bypassing checkpoint restore.
Am I missing something here?
Boris Lublinsky
FDP Architect
boris.lublinsky@lightbend.com
https://www.lightbend.com/
Re: How to run Graph algorithms
Posted by Robert Metzger <rm...@apache.org>.
Hey,
this page explains how to run a Flink job:
https://ci.apache.org/projects/flink/flink-docs-master/getting-started/tutorials/local_setup.html
On Sat, May 25, 2019 at 1:28 PM RAMALINGESWARA RAO THOTTEMPUDI <
trrao@iitkgp.ac.in> wrote:
>
> Respected All,
> I am a new learner of Apache Flink. I want to run existing Graph
> algorithms (examples) given in Flink download software with my own data.
> But I am not getting how to run those existing example algos on my input
> data. Kindly suggest me a solution.
> ------------------------------
> *From: *"bastien dine" <ba...@gmail.com>
> *To: *"Boris Lublinsky" <bo...@lightbend.com>
> *Cc: *"user" <us...@flink.apache.org>
> *Sent: *Saturday, May 25, 2019 1:15:32 PM
> *Subject: *Re: Job cluster and HA
>
> Hello Boris,
> I think you are confused by the name of the shell script
> "standalone-job.sh"
> Which basically means that we start a "standalone job manager" as stated
> in the first comment of
>
> https://github.com/apache/flink/blob/release-1.8/flink-dist/src/main/flink-bin/bin/standalone-job.sh
>
> This is another version of
> : flink-dist/src/main/flink-bin/bin/jobmanager.sh
>
> It's not related to a job
>
> When you configure H-A on a flink cluster, and you submit a job, Flink
> (i.e the jobmanager) store the state of the job in Zookeeper / HDFS
> So when it crashes and comes back (with this entrypoint) it will read in
> ZK / HDFS and restore previous execution
>
> Regards,
> Bastien
>
> ------------------
>
> Bastien DINE
> Data Architect / Software Engineer / Sysadmin
> bastiendine.io
>
>
> Le ven. 24 mai 2019 à 23:22, Boris Lublinsky <
> boris.lublinsky@lightbend.com> a écrit :
>
>> Hi,
>> I was experimenting with HA lately and see that it recovers successfully
>> job, in the case of jobmanager restarts.
>> Now my question is whether it will work for the job cluster.
>> Based on the instructions
>> https://github.com/apache/flink/blob/release-1.8/flink-container/docker/README.md
>> I can see
>> https://github.com/apache/flink/blob/release-1.8/flink-container/docker/docker-entrypoint.sh
>> that
>> In this case the following command is invoked:
>> exec $FLINK_HOME/bin/standalone-job.sh start-foreground "$@“
>>
>> Which means that if a jobManager restarts, the following is going to
>> happen:
>>
>> 1. It will use HA to restore job that was running
>> 2. A new job will be submitted, overwriting restored job and bypassing
>> checkpoint restore.
>>
>> Am I missing something here?
>>
>>
>> Boris Lublinsky
>> FDP Architect
>> boris.lublinsky@lightbend.com
>> https://www.lightbend.com/
>>
>
>
How to run Graph algorithms
Posted by RAMALINGESWARA RAO THOTTEMPUDI <tr...@iitkgp.ac.in>.
Respected All,
I am a new learner of Apache Flink. I want to run existing Graph algorithms (examples) given in Flink download software with my own data. But I am not getting how to run those existing example algos on my input data. Kindly suggest me a solution.
From: "bastien dine" <ba...@gmail.com>
To: "Boris Lublinsky" <bo...@lightbend.com>
Cc: "user" <us...@flink.apache.org>
Sent: Saturday, May 25, 2019 1:15:32 PM
Subject: Re: Job cluster and HA
Hello Boris,
I think you are confused by the name of the shell script "standalone-job.sh"
Which basically means that we start a "standalone job manager" as stated in the first comment of
[ https://github.com/apache/flink/blob/release-1.8/flink-dist/src/main/flink-bin/bin/standalone-job.sh | https://github.com/apache/flink/blob/release-1.8/flink-dist/src/main/flink-bin/bin/standalone-job.sh ]
This is another version of : flink-dist/src/main/flink-bin/bin/jobmanager.sh
It's not related to a job
When you configure H-A on a flink cluster, and you submit a job, Flink (i.e the jobmanager) store the state of the job in Zookeeper / HDFS
So when it crashes and comes back (with this entrypoint) it will read in ZK / HDFS and restore previous execution
Regards,
Bastien
------------------
Bastien DINE
Data Architect / Software Engineer / Sysadmin
[ http://bastiendine.io/ | bastiendine.io ]
Le ven. 24 mai 2019 à 23:22, Boris Lublinsky < [ mailto:boris.lublinsky@lightbend.com | boris.lublinsky@lightbend.com ] > a écrit :
Hi,
I was experimenting with HA lately and see that it recovers successfully job, in the case of jobmanager restarts.
Now my question is whether it will work for the job cluster.
Based on the instructions [ https://github.com/apache/flink/blob/release-1.8/flink-container/docker/README.md | https://github.com/apache/flink/blob/release-1.8/flink-container/docker/README.md ]
I can see [ https://github.com/apache/flink/blob/release-1.8/flink-container/docker/docker-entrypoint.sh | https://github.com/apache/flink/blob/release-1.8/flink-container/docker/docker-entrypoint.sh ] that
In this case the following command is invoked:
exec $FLINK_HOME /bin/standalone-job.sh start-foreground " $@ “
Which means that if a jobManager restarts, the following is going to happen:
1. It will use HA to restore job that was running
2. A new job will be submitted, overwriting restored job and bypassing checkpoint restore.
Am I missing something here?
Boris Lublinsky
FDP Architect
[ mailto:boris.lublinsky@lightbend.com | boris.lublinsky@lightbend.com ]
[ https://www.lightbend.com/ | https://www.lightbend.com/ ]
Re: Job cluster and HA
Posted by Boris Lublinsky <bo...@lightbend.com>.
I understand this.
What I am not sure about is the sequence:
It will come back and restore from ZK/HDFS
And then it will try to start a job specified in the class, right?
Which will overwrite (potentially) the restored job.
How does it know not to start the job defined in the class, once the previous one was restored?
Boris Lublinsky
FDP Architect
boris.lublinsky@lightbend.com
https://www.lightbend.com/
> On May 25, 2019, at 2:45 AM, bastien dine <ba...@gmail.com> wrote:
>
> Hello Boris,
>
> I think you are confused by the name of the shell script "standalone-job.sh"
> Which basically means that we start a "standalone job manager" as stated in the first comment of
> https://github.com/apache/flink/blob/release-1.8/flink-dist/src/main/flink-bin/bin/standalone-job.sh <https://github.com/apache/flink/blob/release-1.8/flink-dist/src/main/flink-bin/bin/standalone-job.sh>
> This is another version of : flink-dist/src/main/flink-bin/bin/jobmanager.sh
>
> It's not related to a job
>
> When you configure H-A on a flink cluster, and you submit a job, Flink (i.e the jobmanager) store the state of the job in Zookeeper / HDFS
> So when it crashes and comes back (with this entrypoint) it will read in ZK / HDFS and restore previous execution
>
> Regards,
> Bastien
>
> ------------------
>
> Bastien DINE
> Data Architect / Software Engineer / Sysadmin
> bastiendine.io <http://bastiendine.io/>
>
>
> Le ven. 24 mai 2019 à 23:22, Boris Lublinsky <boris.lublinsky@lightbend.com <ma...@lightbend.com>> a écrit :
> Hi,
> I was experimenting with HA lately and see that it recovers successfully job, in the case of jobmanager restarts.
> Now my question is whether it will work for the job cluster.
> Based on the instructions https://github.com/apache/flink/blob/release-1.8/flink-container/docker/README.md <https://github.com/apache/flink/blob/release-1.8/flink-container/docker/README.md>
> I can see https://github.com/apache/flink/blob/release-1.8/flink-container/docker/docker-entrypoint.sh <https://github.com/apache/flink/blob/release-1.8/flink-container/docker/docker-entrypoint.sh> that
> In this case the following command is invoked:
> exec $FLINK_HOME/bin/standalone-job.sh start-foreground "$@“
>
> Which means that if a jobManager restarts, the following is going to happen:
>
> 1. It will use HA to restore job that was running
> 2. A new job will be submitted, overwriting restored job and bypassing checkpoint restore.
>
> Am I missing something here?
>
>
> Boris Lublinsky
> FDP Architect
> boris.lublinsky@lightbend.com <ma...@lightbend.com>
> https://www.lightbend.com/ <https://www.lightbend.com/>
Re: Job cluster and HA
Posted by bastien dine <ba...@gmail.com>.
Hello Boris,
I think you are confused by the name of the shell script "standalone-job.sh"
Which basically means that we start a "standalone job manager" as stated in
the first comment of
https://github.com/apache/flink/blob/release-1.8/flink-dist/src/main/flink-bin/bin/standalone-job.sh
This is another version of : flink-dist/src/main/flink-bin/bin/jobmanager.sh
It's not related to a job
When you configure H-A on a flink cluster, and you submit a job, Flink (i.e
the jobmanager) store the state of the job in Zookeeper / HDFS
So when it crashes and comes back (with this entrypoint) it will read in ZK
/ HDFS and restore previous execution
Regards,
Bastien
------------------
Bastien DINE
Data Architect / Software Engineer / Sysadmin
bastiendine.io
Le ven. 24 mai 2019 à 23:22, Boris Lublinsky <bo...@lightbend.com>
a écrit :
> Hi,
> I was experimenting with HA lately and see that it recovers successfully
> job, in the case of jobmanager restarts.
> Now my question is whether it will work for the job cluster.
> Based on the instructions
> https://github.com/apache/flink/blob/release-1.8/flink-container/docker/README.md
> I can see
> https://github.com/apache/flink/blob/release-1.8/flink-container/docker/docker-entrypoint.sh
> that
> In this case the following command is invoked:
> exec $FLINK_HOME/bin/standalone-job.sh start-foreground "$@“
>
> Which means that if a jobManager restarts, the following is going to
> happen:
>
> 1. It will use HA to restore job that was running
> 2. A new job will be submitted, overwriting restored job and bypassing
> checkpoint restore.
>
> Am I missing something here?
>
>
> Boris Lublinsky
> FDP Architect
> boris.lublinsky@lightbend.com
> https://www.lightbend.com/
>
>