You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Emmanuel <el...@msn.com> on 2015/09/09 01:59:24 UTC
Flink HA mode
Looking at Flink HA mode.
Why do you need to have the list of masters in the config if zookeeper is used to keep track of them? In an environment like Google Cloud or Container Engine, the JM may come back up but will likely have another IP address.
Is the masters config file only for bootstrapping or is it effectively to keep track of JM nodes? Is it what TM nodes use to know their JobManager? Or do they query Zookeeper?
In the scenario above (2 JMs, one fails, and comes back up as another instance, with different IP) how does the JM join the cluster and how do TMs know about the new JM?
ThanksE
Re: Flink HA mode
Posted by Ufuk Celebi <uc...@apache.org>.
Hey Emmanuel,
as Fabian said it's currently under development in 0.10-SNAPSHOT. It
doesn't recover jobs yet (wait a few more days for this), but the
infrastructure for leader election, starting ZooKeeper, etc. is in place.
Comments are very welcome if you find time to play around with it. If you
like, I can ping you when the submitted job recovery is merged as well.
– Ufuk
On Thu, Sep 10, 2015 at 6:54 AM, Fabian Hueske <fh...@gmail.com> wrote:
> Hi Emmanuel,
> yes Master HA is currently under development and only available in 0.10
> snapshot. AFAIK, it is almost but not completely done yet.
>
> Best, Fabian
> On Sep 10, 2015 01:29, "Emmanuel" <el...@msn.com> wrote:
>
>> is this a 0.10 snapshot feature only? I'm using 0.9.1 right now
>>
>>
>> ------------------------------
>> From: eleroy@msn.com
>> To: user@flink.apache.org
>> Subject: RE: Flink HA mode
>> Date: Wed, 9 Sep 2015 16:11:38 -0700
>>
>> Been playing with the HA...
>> I find the UIs confusing here:
>> in the dashboard on one side I see 0 slots 0 taskmanagers, but a job
>> running, while on the other side I see my taskmanagers and slots but no
>> jobs...
>> putting the UI being a proxy, it's load balanced to the JM, so I can't
>> tell which is which (my bad) but how would I be able to tell which is the
>> leader anyway? Do I need to query ZK to know?
>>
>> Looking at ZK, I actually don't see any keys for Flink... so I'm
>> wondering if this is working as expected.
>> The config lists the options
>>
>> recovery.mode: zookeeper
>> ha.zookeeper.quorum: zookeeper:2181
>>
>> but I don't see any mention of Zookeeper in the JM logs.
>>
>> I start the JMs and TMs manually with
>> ./jobmanager.sh start streaming
>> ./taskmanager.sh start streaming
>>
>> so i guess there is no need for the masters file.
>>
>> What else am I missing here?
>> Do I need to list the multiple JMs IPs in the jobmanager.rpc.address?
>>
>> Thanks
>>
>> ------------------------------
>> Date: Wed, 9 Sep 2015 10:19:36 +0200
>> Subject: Re: Flink HA mode
>> From: trohrmann@apache.org
>> To: user@flink.apache.org
>>
>> The only necessary information for the JobManager and TaskManager is to
>> know where to find the ZooKeeper quorum to do leader election and retrieve
>> the leader address from. This will be configured via the config parameter
>> `ha.zookeeper.quorum`.
>>
>> On Wed, Sep 9, 2015 at 10:15 AM, Stephan Ewen <se...@apache.org> wrote:
>>
>> TL;DR is that you are right, it is only the initial list. If a JobManager
>> comes back with a new IP address, it will be available.
>>
>> On Wed, Sep 9, 2015 at 8:35 AM, Ufuk Celebi <uc...@apache.org> wrote:
>>
>>
>> > On 09 Sep 2015, at 04:48, Emmanuel <el...@msn.com> wrote:
>> >
>> > my questions is: how critical is the bootstrap ip list in masters?
>>
>> Hey Emmanuel,
>>
>> good questions. I read over the docs for this again [1] and you are right
>> that we should make this clearer.
>>
>> The “masters" file is only relevant for the start/stop cluster scripts
>> (Flink standalone mode).
>>
>> If you specify hosts in the “masters" file the start-cluster scripts will
>> use these hosts to start job managers. After that all coordination happens
>> via ZooKeeper via a leader election and retrieval service. All job managers
>> elect a single leader and task managers and clients (submitting programs)
>> retrieve this leader via ZooKeeper. If a job manager fails and becomes
>> available again, it will publish itself via this mechanism (if it becomes
>> leader at some point again). There was a recent PR [2] which introduced
>> this. You can read over the very good PR description for more info for now.
>>
>> [1]
>> https://ci.apache.org/projects/flink/flink-docs-master/setup/jobmanager_high_availability.html
>>
>> [2] https://github.com/apache/flink/pull/1016
>>
>>
>> > does this get updated or does it have to be updated by some other
>> service?
>>
>> If you start a new cluster on GCE with different hosts and use Flink’s
>> standalone mode you have to set this up again. This is the same for the
>> “slaves” file.
>>
>>
>> Does this answer your question? If anything is unclear, please post here.
>> :)
>>
>> – Ufuk
>>
>>
>>
>>
RE: Flink HA mode
Posted by Fabian Hueske <fh...@gmail.com>.
Hi Emmanuel,
yes Master HA is currently under development and only available in 0.10
snapshot. AFAIK, it is almost but not completely done yet.
Best, Fabian
On Sep 10, 2015 01:29, "Emmanuel" <el...@msn.com> wrote:
> is this a 0.10 snapshot feature only? I'm using 0.9.1 right now
>
>
> ------------------------------
> From: eleroy@msn.com
> To: user@flink.apache.org
> Subject: RE: Flink HA mode
> Date: Wed, 9 Sep 2015 16:11:38 -0700
>
> Been playing with the HA...
> I find the UIs confusing here:
> in the dashboard on one side I see 0 slots 0 taskmanagers, but a job
> running, while on the other side I see my taskmanagers and slots but no
> jobs...
> putting the UI being a proxy, it's load balanced to the JM, so I can't
> tell which is which (my bad) but how would I be able to tell which is the
> leader anyway? Do I need to query ZK to know?
>
> Looking at ZK, I actually don't see any keys for Flink... so I'm wondering
> if this is working as expected.
> The config lists the options
>
> recovery.mode: zookeeper
> ha.zookeeper.quorum: zookeeper:2181
>
> but I don't see any mention of Zookeeper in the JM logs.
>
> I start the JMs and TMs manually with
> ./jobmanager.sh start streaming
> ./taskmanager.sh start streaming
>
> so i guess there is no need for the masters file.
>
> What else am I missing here?
> Do I need to list the multiple JMs IPs in the jobmanager.rpc.address?
>
> Thanks
>
> ------------------------------
> Date: Wed, 9 Sep 2015 10:19:36 +0200
> Subject: Re: Flink HA mode
> From: trohrmann@apache.org
> To: user@flink.apache.org
>
> The only necessary information for the JobManager and TaskManager is to
> know where to find the ZooKeeper quorum to do leader election and retrieve
> the leader address from. This will be configured via the config parameter
> `ha.zookeeper.quorum`.
>
> On Wed, Sep 9, 2015 at 10:15 AM, Stephan Ewen <se...@apache.org> wrote:
>
> TL;DR is that you are right, it is only the initial list. If a JobManager
> comes back with a new IP address, it will be available.
>
> On Wed, Sep 9, 2015 at 8:35 AM, Ufuk Celebi <uc...@apache.org> wrote:
>
>
> > On 09 Sep 2015, at 04:48, Emmanuel <el...@msn.com> wrote:
> >
> > my questions is: how critical is the bootstrap ip list in masters?
>
> Hey Emmanuel,
>
> good questions. I read over the docs for this again [1] and you are right
> that we should make this clearer.
>
> The “masters" file is only relevant for the start/stop cluster scripts
> (Flink standalone mode).
>
> If you specify hosts in the “masters" file the start-cluster scripts will
> use these hosts to start job managers. After that all coordination happens
> via ZooKeeper via a leader election and retrieval service. All job managers
> elect a single leader and task managers and clients (submitting programs)
> retrieve this leader via ZooKeeper. If a job manager fails and becomes
> available again, it will publish itself via this mechanism (if it becomes
> leader at some point again). There was a recent PR [2] which introduced
> this. You can read over the very good PR description for more info for now.
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-master/setup/jobmanager_high_availability.html
>
> [2] https://github.com/apache/flink/pull/1016
>
>
> > does this get updated or does it have to be updated by some other
> service?
>
> If you start a new cluster on GCE with different hosts and use Flink’s
> standalone mode you have to set this up again. This is the same for the
> “slaves” file.
>
>
> Does this answer your question? If anything is unclear, please post here.
> :)
>
> – Ufuk
>
>
>
>
RE: Flink HA mode
Posted by Emmanuel <el...@msn.com>.
is this a 0.10 snapshot feature only? I'm using 0.9.1 right now
From: eleroy@msn.com
To: user@flink.apache.org
Subject: RE: Flink HA mode
Date: Wed, 9 Sep 2015 16:11:38 -0700
Been playing with the HA...I find the UIs confusing here: in the dashboard on one side I see 0 slots 0 taskmanagers, but a job running, while on the other side I see my taskmanagers and slots but no jobs... putting the UI being a proxy, it's load balanced to the JM, so I can't tell which is which (my bad) but how would I be able to tell which is the leader anyway? Do I need to query ZK to know?
Looking at ZK, I actually don't see any keys for Flink... so I'm wondering if this is working as expected.The config lists the optionsrecovery.mode: zookeeper
ha.zookeeper.quorum: zookeeper:2181but I don't see any mention of Zookeeper in the JM logs.
I start the JMs and TMs manually with ./jobmanager.sh start streaming./taskmanager.sh start streaming
so i guess there is no need for the masters file.
What else am I missing here?Do I need to list the multiple JMs IPs in the jobmanager.rpc.address?
Thanks
Date: Wed, 9 Sep 2015 10:19:36 +0200
Subject: Re: Flink HA mode
From: trohrmann@apache.org
To: user@flink.apache.org
The only necessary information for the JobManager and TaskManager is to know where to find the ZooKeeper quorum to do leader election and retrieve the leader address from. This will be configured via the config parameter `ha.zookeeper.quorum`.
On Wed, Sep 9, 2015 at 10:15 AM, Stephan Ewen <se...@apache.org> wrote:
TL;DR is that you are right, it is only the initial list. If a JobManager comes back with a new IP address, it will be available.
On Wed, Sep 9, 2015 at 8:35 AM, Ufuk Celebi <uc...@apache.org> wrote:
> On 09 Sep 2015, at 04:48, Emmanuel <el...@msn.com> wrote:
>
> my questions is: how critical is the bootstrap ip list in masters?
Hey Emmanuel,
good questions. I read over the docs for this again [1] and you are right that we should make this clearer.
The “masters" file is only relevant for the start/stop cluster scripts (Flink standalone mode).
If you specify hosts in the “masters" file the start-cluster scripts will use these hosts to start job managers. After that all coordination happens via ZooKeeper via a leader election and retrieval service. All job managers elect a single leader and task managers and clients (submitting programs) retrieve this leader via ZooKeeper. If a job manager fails and becomes available again, it will publish itself via this mechanism (if it becomes leader at some point again). There was a recent PR [2] which introduced this. You can read over the very good PR description for more info for now.
[1] https://ci.apache.org/projects/flink/flink-docs-master/setup/jobmanager_high_availability.html
[2] https://github.com/apache/flink/pull/1016
> does this get updated or does it have to be updated by some other service?
If you start a new cluster on GCE with different hosts and use Flink’s standalone mode you have to set this up again. This is the same for the “slaves” file.
Does this answer your question? If anything is unclear, please post here. :)
– Ufuk
RE: Flink HA mode
Posted by Emmanuel <el...@msn.com>.
Been playing with the HA...I find the UIs confusing here: in the dashboard on one side I see 0 slots 0 taskmanagers, but a job running, while on the other side I see my taskmanagers and slots but no jobs... putting the UI being a proxy, it's load balanced to the JM, so I can't tell which is which (my bad) but how would I be able to tell which is the leader anyway? Do I need to query ZK to know?
Looking at ZK, I actually don't see any keys for Flink... so I'm wondering if this is working as expected.The config lists the optionsrecovery.mode: zookeeper
ha.zookeeper.quorum: zookeeper:2181but I don't see any mention of Zookeeper in the JM logs.
I start the JMs and TMs manually with ./jobmanager.sh start streaming./taskmanager.sh start streaming
so i guess there is no need for the masters file.
What else am I missing here?Do I need to list the multiple JMs IPs in the jobmanager.rpc.address?
Thanks
Date: Wed, 9 Sep 2015 10:19:36 +0200
Subject: Re: Flink HA mode
From: trohrmann@apache.org
To: user@flink.apache.org
The only necessary information for the JobManager and TaskManager is to know where to find the ZooKeeper quorum to do leader election and retrieve the leader address from. This will be configured via the config parameter `ha.zookeeper.quorum`.
On Wed, Sep 9, 2015 at 10:15 AM, Stephan Ewen <se...@apache.org> wrote:
TL;DR is that you are right, it is only the initial list. If a JobManager comes back with a new IP address, it will be available.
On Wed, Sep 9, 2015 at 8:35 AM, Ufuk Celebi <uc...@apache.org> wrote:
> On 09 Sep 2015, at 04:48, Emmanuel <el...@msn.com> wrote:
>
> my questions is: how critical is the bootstrap ip list in masters?
Hey Emmanuel,
good questions. I read over the docs for this again [1] and you are right that we should make this clearer.
The “masters" file is only relevant for the start/stop cluster scripts (Flink standalone mode).
If you specify hosts in the “masters" file the start-cluster scripts will use these hosts to start job managers. After that all coordination happens via ZooKeeper via a leader election and retrieval service. All job managers elect a single leader and task managers and clients (submitting programs) retrieve this leader via ZooKeeper. If a job manager fails and becomes available again, it will publish itself via this mechanism (if it becomes leader at some point again). There was a recent PR [2] which introduced this. You can read over the very good PR description for more info for now.
[1] https://ci.apache.org/projects/flink/flink-docs-master/setup/jobmanager_high_availability.html
[2] https://github.com/apache/flink/pull/1016
> does this get updated or does it have to be updated by some other service?
If you start a new cluster on GCE with different hosts and use Flink’s standalone mode you have to set this up again. This is the same for the “slaves” file.
Does this answer your question? If anything is unclear, please post here. :)
– Ufuk
Re: Flink HA mode
Posted by Till Rohrmann <tr...@apache.org>.
The only necessary information for the JobManager and TaskManager is to
know where to find the ZooKeeper quorum to do leader election and retrieve
the leader address from. This will be configured via the config parameter
`ha.zookeeper.quorum`.
On Wed, Sep 9, 2015 at 10:15 AM, Stephan Ewen <se...@apache.org> wrote:
> TL;DR is that you are right, it is only the initial list. If a JobManager
> comes back with a new IP address, it will be available.
>
> On Wed, Sep 9, 2015 at 8:35 AM, Ufuk Celebi <uc...@apache.org> wrote:
>
>>
>> > On 09 Sep 2015, at 04:48, Emmanuel <el...@msn.com> wrote:
>> >
>> > my questions is: how critical is the bootstrap ip list in masters?
>>
>> Hey Emmanuel,
>>
>> good questions. I read over the docs for this again [1] and you are right
>> that we should make this clearer.
>>
>> The “masters" file is only relevant for the start/stop cluster scripts
>> (Flink standalone mode).
>>
>> If you specify hosts in the “masters" file the start-cluster scripts will
>> use these hosts to start job managers. After that all coordination happens
>> via ZooKeeper via a leader election and retrieval service. All job managers
>> elect a single leader and task managers and clients (submitting programs)
>> retrieve this leader via ZooKeeper. If a job manager fails and becomes
>> available again, it will publish itself via this mechanism (if it becomes
>> leader at some point again). There was a recent PR [2] which introduced
>> this. You can read over the very good PR description for more info for now.
>>
>> [1]
>> https://ci.apache.org/projects/flink/flink-docs-master/setup/jobmanager_high_availability.html
>>
>> [2] https://github.com/apache/flink/pull/1016
>>
>>
>> > does this get updated or does it have to be updated by some other
>> service?
>>
>> If you start a new cluster on GCE with different hosts and use Flink’s
>> standalone mode you have to set this up again. This is the same for the
>> “slaves” file.
>>
>>
>> Does this answer your question? If anything is unclear, please post here.
>> :)
>>
>> – Ufuk
>>
>>
>
Re: Flink HA mode
Posted by Stephan Ewen <se...@apache.org>.
TL;DR is that you are right, it is only the initial list. If a JobManager
comes back with a new IP address, it will be available.
On Wed, Sep 9, 2015 at 8:35 AM, Ufuk Celebi <uc...@apache.org> wrote:
>
> > On 09 Sep 2015, at 04:48, Emmanuel <el...@msn.com> wrote:
> >
> > my questions is: how critical is the bootstrap ip list in masters?
>
> Hey Emmanuel,
>
> good questions. I read over the docs for this again [1] and you are right
> that we should make this clearer.
>
> The “masters" file is only relevant for the start/stop cluster scripts
> (Flink standalone mode).
>
> If you specify hosts in the “masters" file the start-cluster scripts will
> use these hosts to start job managers. After that all coordination happens
> via ZooKeeper via a leader election and retrieval service. All job managers
> elect a single leader and task managers and clients (submitting programs)
> retrieve this leader via ZooKeeper. If a job manager fails and becomes
> available again, it will publish itself via this mechanism (if it becomes
> leader at some point again). There was a recent PR [2] which introduced
> this. You can read over the very good PR description for more info for now.
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-master/setup/jobmanager_high_availability.html
>
> [2] https://github.com/apache/flink/pull/1016
>
>
> > does this get updated or does it have to be updated by some other
> service?
>
> If you start a new cluster on GCE with different hosts and use Flink’s
> standalone mode you have to set this up again. This is the same for the
> “slaves” file.
>
>
> Does this answer your question? If anything is unclear, please post here.
> :)
>
> – Ufuk
>
>
Re: Flink HA mode
Posted by Ufuk Celebi <uc...@apache.org>.
> On 09 Sep 2015, at 04:48, Emmanuel <el...@msn.com> wrote:
>
> my questions is: how critical is the bootstrap ip list in masters?
Hey Emmanuel,
good questions. I read over the docs for this again [1] and you are right that we should make this clearer.
The “masters" file is only relevant for the start/stop cluster scripts (Flink standalone mode).
If you specify hosts in the “masters" file the start-cluster scripts will use these hosts to start job managers. After that all coordination happens via ZooKeeper via a leader election and retrieval service. All job managers elect a single leader and task managers and clients (submitting programs) retrieve this leader via ZooKeeper. If a job manager fails and becomes available again, it will publish itself via this mechanism (if it becomes leader at some point again). There was a recent PR [2] which introduced this. You can read over the very good PR description for more info for now.
[1] https://ci.apache.org/projects/flink/flink-docs-master/setup/jobmanager_high_availability.html
[2] https://github.com/apache/flink/pull/1016
> does this get updated or does it have to be updated by some other service?
If you start a new cluster on GCE with different hosts and use Flink’s standalone mode you have to set this up again. This is the same for the “slaves” file.
Does this answer your question? If anything is unclear, please post here. :)
– Ufuk
RE: Flink HA mode
Posted by Emmanuel <el...@msn.com>.
my questions is: how critical is the bootstrap ip list in masters? does this get updated or does it have to be updated by some other service?
From: zhangrucong@huawei.com
To: user@flink.apache.org
Subject: re: Flink HA mode
Date: Wed, 9 Sep 2015 00:48:42 +0000
In order to discover new JM,I think must use ZK. ZK has the ability to find a new node or the content of node changed.
First JM must create node in ZK, and write IP and port in node. TMs watch this node. When TMs find the node content change, TMs reconnect the new
JM.
Thanks.
发件人: Emmanuel [mailto:eleroy@msn.com]
发送时间: 2015年9月9日
7:59
收件人: user@flink.apache.org
主题: Flink HA mode
Looking at Flink HA mode.
Why do you need to have the list of masters in the config if zookeeper is used to keep track of them?
In an environment like Google Cloud or Container Engine, the JM may come back up but will likely have another IP address.
Is the masters config file only for bootstrapping or is it effectively to keep track of JM nodes?
Is it what TM nodes use to know their JobManager? Or do they query Zookeeper?
In the scenario above (2 JMs, one fails, and comes back up as another instance, with different IP) how does the JM join the cluster and how do TMs know about the new JM?
Thanks
E
re: Flink HA mode
Posted by Zhangrucong <zh...@huawei.com>.
In order to discover new JM,I think must use ZK. ZK has the ability to find a new node or the content of node changed.
First JM must create node in ZK, and write IP and port in node. TMs watch this node. When TMs find the node content change, TMs reconnect the new JM.
Thanks.
发件人: Emmanuel [mailto:eleroy@msn.com]
发送时间: 2015年9月9日 7:59
收件人: user@flink.apache.org
主题: Flink HA mode
Looking at Flink HA mode.
Why do you need to have the list of masters in the config if zookeeper is used to keep track of them?
In an environment like Google Cloud or Container Engine, the JM may come back up but will likely have another IP address.
Is the masters config file only for bootstrapping or is it effectively to keep track of JM nodes?
Is it what TM nodes use to know their JobManager? Or do they query Zookeeper?
In the scenario above (2 JMs, one fails, and comes back up as another instance, with different IP) how does the JM join the cluster and how do TMs know about the new JM?
Thanks
E