You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by Lei Gao <lg...@linkedin.com> on 2010/04/30 20:56:11 UTC
Question on maintaining leader/membership status in zookeeper
Hi,
I have a general question on how zookeeper can maintain its view of the user cluster (that zookeeper manages) that is consistent with the nodes in the user cluster. In other words, when zookeeper considers the current leader is unavailable, does it really guarantee that a majority of nodes in the user cluster can’t reach the current leader? The same question applies to the membership service as well. Because the zookeeper can be partitioned from a majority of the nodes in the user cluster. How does the zookeeper handle situations like this?
Thanks,
Lei
Re: Question on maintaining leader/membership status in zookeeper
Posted by Ted Dunning <te...@gmail.com>.
Lei,
I think that Mahadev was correct that there is some confusion here.
Leader election is normally a term used for an operation that is entirely
internal to ZK. It is very robust and you probably don't need to worry
about it.
You can then use ZK in your application to pick a lead machine for other
operations. In that case, essentially every failure scenario is handled by
the standard recipe. In your example where the master and slave are cut
off, but both still have access to ZK, all that will happen is that the
master cannot communicate with the slave. Both will still be clear about
who is in which role.
The case where the master is cut off from both ZK and the slave is also
handled well as is the case where the master is cut off from ZK, but not
from the slave. In both cases, the master will get a connection loss event
and stop trying to act like a master and the slave will be notified that the
master has dropped out of its role.
On Fri, Apr 30, 2010 at 4:05 PM, Lei Gao <lg...@linkedin.com> wrote:
> Hi Mahadev,
>
> Why would the leader be disconnected from ZK? ZK is fine communicating with
> the leader in this case. We are talking about asymmetric network failure.
> Yes. Leader could consider all the slaves being down if it tracks the
> status
> of all slaves himself. But I guess if ZK is used for for membership
> management, neither the leader nor the slaves will be considered
> disconnected because they can all connect to ZK.
>
> Thanks,
>
> Lei
>
>
> On 4/30/10 3:47 PM, "Mahadev Konar" <ma...@yahoo-inc.com> wrote:
>
> > Hi Lei,
> >
> > In this case, the Leader will be disconnected from ZK cluster and will
> give
> > up its leadership. Since its disconnected, ZK cluster will realize that
> the
> > Leader is dead!....
> >
> > When Zk cluster realizes that the Leader is dead (this is because the zk
> > cluster hasn't heard from the Leader for a certain time.... Configurable
> via
> > session timeout parameter), the slaves will be notified of this via
> watchers
> > in zookeeper cluster. The slaves will realize that the Leader is gone and
> > will relect a new Leader and will start working with the new Leader.
> >
> > Does that answer your question?
> >
> > You might want to look though the documentation of ZK to understand its
> use
> > case and how it solves these kind of issues....
> >
> > Thanks
> > mahadev
> >
> >
> > On 4/30/10 2:08 PM, "Lei Gao" <lg...@linkedin.com> wrote:
> >
> >> Thank you all for your answers. It clarifies a lot of my confusion about
> the
> >> service guarantees of ZK. I am still struggling with one failure case (I
> am
> >> not trying to be the pain in the neck. But I need to have a full
> >> understanding of what ZK can offer before I make a decision on whether
> to
> >> used it in my cluster.)
> >>
> >> Assume the following topology:
> >>
> >> Leader ==== ZK cluster
> >> \\ //
> >> \\ //
> >> \\ //
> >> Slave(s)
> >>
> >> If I am asymmetric network failure such that the connection between
> Leader
> >> and Slave(s) are broken while all other connections are still alive,
> would
> >> my system hang after some point? Because no new leader election will be
> >> initiated by slaves and the leader can't get the work to slave(s).
> >>
> >> Thanks,
> >>
> >> Lei
> >>
> >> On 4/30/10 1:54 PM, "Ted Dunning" <te...@gmail.com> wrote:
> >>
> >>> If one of your user clients can no longer reach one member of the ZK
> >>> cluster, then it will try to reach another. If it succeeds, then it
> will
> >>> continue without any problems as long as the ZK cluster itself is OK.
> >>>
> >>> This applies for all the ZK recipes. You will have to be a little bit
> >>> careful to handle connection loss, but that should get easier soon (and
> >>> isn't all that difficult anyway).
> >>>
> >>> On Fri, Apr 30, 2010 at 1:26 PM, Lei Gao <lg...@linkedin.com> wrote:
> >>>
> >>>> I am not talking about the leader election within zookeeper cluster. I
> >>>> guess
> >>>> I didn't make the discussion context clear. In my case, I run a
> cluster
> >>>> that
> >>>> uses zookeeper for doing the leader election. Yes, nodes in my cluster
> are
> >>>> the clients of zookeeper. Those nodes depend on zookeeper to elect a
> new
> >>>> leader and figure out what the current leader is. So if the zookeeper
> >>>> (think
> >>>> of it as a stand-alone entity) becomes unavailabe in the way I've
> described
> >>>> earlier, how can I handle such situation so my cluster can still
> function
> >>>> while a majority of nodes still connect to each other (but not to the
> >>>> zookeeper)?
> >>>>
> >>
> >
>
>
Re: Question on maintaining leader/membership status in zookeeper
Posted by Mahadev Konar <ma...@yahoo-inc.com>.
HI Lei,
ZooKeeper provides a set of primitives which allows you to do all kinds of
things! You might want to take a look at the api and some examples of
zookeeper recipes to see how it works and probably that will clear things
out for you.
Here are the links:
http://hadoop.apache.org/zookeeper/docs/r3.3.0/recipes.html
Thanks
mahadev
On 4/30/10 4:46 PM, "Lei Gao" <lg...@linkedin.com> wrote:
> Hi Mahadev,
>
> First of all, I like to thank you for being patient with me - my questions
> seem unclear to many of you who try to help me.
>
> I guess clients have to be smart enough to trigger a new leader election by
> trying to delete the znode. But in this case, ZK should not allow any single
> or multiple (as long as they are less than a quorum) client(s) to delete the
> znode responding to the master, right? A new consensus among clients (NOT
> among the nodes in zk cluster) has to be there for the znode to be deleted,
> right? Does zk have this capability or the clients have to come to this
> consensus outside of zk before trying to delete the znode in zk?
>
> Thanks,
>
> Lei
>
>> Hi Lei,
>> Sorry I minsinterpreted your question! The scenario you describe could be
>> handled in such a way -
>>
>> You could have a status node in ZooKeeper which every slave will subscribe
>> to and update! If one of the slave nodes sees that there have been too many
>> connection refused to the Leader by the slaves, the slave could go ahead and
>> delete the Leader znode, and force the Leader to give up its leadership. I
>> am not describing a deatiled way to do it, but its not very hard to come up
>> with a design for this.
>>
>>
>>
>> Do you intend to have the Leader and Slaves in different Network (different
>> ACLs I mean) protected zones? In that case, it is a legitimate concern else
>> I do think assymetric network partition would be very unlikely to happen.
>>
>> Do you usually see network partitions in such scenarios?
>>
>> Thanks
>> mahadev
>>
>>
>> On 4/30/10 4:05 PM, "Lei Gao" <lg...@linkedin.com> wrote:
>>
>>> Hi Mahadev,
>>>
>>> Why would the leader be disconnected from ZK? ZK is fine communicating with
>>> the leader in this case. We are talking about asymmetric network failure.
>>> Yes. Leader could consider all the slaves being down if it tracks the status
>>> of all slaves himself. But I guess if ZK is used for for membership
>>> management, neither the leader nor the slaves will be considered
>>> disconnected because they can all connect to ZK.
>>>
>>> Thanks,
>>>
>>> Lei
>>>
>>>
>>> On 4/30/10 3:47 PM, "Mahadev Konar" <ma...@yahoo-inc.com> wrote:
>>>
>>>> Hi Lei,
>>>>
>>>> In this case, the Leader will be disconnected from ZK cluster and will give
>>>> up its leadership. Since its disconnected, ZK cluster will realize that the
>>>> Leader is dead!....
>>>>
>>>> When Zk cluster realizes that the Leader is dead (this is because the zk
>>>> cluster hasn't heard from the Leader for a certain time.... Configurable
>>>> via
>>>> session timeout parameter), the slaves will be notified of this via
>>>> watchers
>>>> in zookeeper cluster. The slaves will realize that the Leader is gone and
>>>> will relect a new Leader and will start working with the new Leader.
>>>>
>>>> Does that answer your question?
>>>>
>>>> You might want to look though the documentation of ZK to understand its use
>>>> case and how it solves these kind of issues....
>>>>
>>>> Thanks
>>>> mahadev
>>>>
>>>>
>>>> On 4/30/10 2:08 PM, "Lei Gao" <lg...@linkedin.com> wrote:
>>>>
>>>>> Thank you all for your answers. It clarifies a lot of my confusion about
>>>>> the
>>>>> service guarantees of ZK. I am still struggling with one failure case (I
>>>>> am
>>>>> not trying to be the pain in the neck. But I need to have a full
>>>>> understanding of what ZK can offer before I make a decision on whether to
>>>>> used it in my cluster.)
>>>>>
>>>>> Assume the following topology:
>>>>>
>>>>> Leader ==== ZK cluster
>>>>> \\ //
>>>>> \\ //
>>>>> \\ //
>>>>> Slave(s)
>>>>>
>>>>> If I am asymmetric network failure such that the connection between Leader
>>>>> and Slave(s) are broken while all other connections are still alive, would
>>>>> my system hang after some point? Because no new leader election will be
>>>>> initiated by slaves and the leader can't get the work to slave(s).
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Lei
>>>>>
>>>>> On 4/30/10 1:54 PM, "Ted Dunning" <te...@gmail.com> wrote:
>>>>>
>>>>>> If one of your user clients can no longer reach one member of the ZK
>>>>>> cluster, then it will try to reach another. If it succeeds, then it will
>>>>>> continue without any problems as long as the ZK cluster itself is OK.
>>>>>>
>>>>>> This applies for all the ZK recipes. You will have to be a little bit
>>>>>> careful to handle connection loss, but that should get easier soon (and
>>>>>> isn't all that difficult anyway).
>>>>>>
>>>>>> On Fri, Apr 30, 2010 at 1:26 PM, Lei Gao <lg...@linkedin.com> wrote:
>>>>>>
>>>>>>> I am not talking about the leader election within zookeeper cluster. I
>>>>>>> guess
>>>>>>> I didn't make the discussion context clear. In my case, I run a cluster
>>>>>>> that
>>>>>>> uses zookeeper for doing the leader election. Yes, nodes in my cluster
>>>>>>> are
>>>>>>> the clients of zookeeper. Those nodes depend on zookeeper to elect a
>>>>>>> new
>>>>>>> leader and figure out what the current leader is. So if the zookeeper
>>>>>>> (think
>>>>>>> of it as a stand-alone entity) becomes unavailabe in the way I've
>>>>>>> described
>>>>>>> earlier, how can I handle such situation so my cluster can still
>>>>>>> function
>>>>>>> while a majority of nodes still connect to each other (but not to the
>>>>>>> zookeeper)?
>>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Question on maintaining leader/membership status in zookeeper
Posted by Lei Gao <lg...@linkedin.com>.
Hi Mahadev,
First of all, I like to thank you for being patient with me - my questions
seem unclear to many of you who try to help me.
I guess clients have to be smart enough to trigger a new leader election by
trying to delete the znode. But in this case, ZK should not allow any single
or multiple (as long as they are less than a quorum) client(s) to delete the
znode responding to the master, right? A new consensus among clients (NOT
among the nodes in zk cluster) has to be there for the znode to be deleted,
right? Does zk have this capability or the clients have to come to this
consensus outside of zk before trying to delete the znode in zk?
Thanks,
Lei
> Hi Lei,
> Sorry I minsinterpreted your question! The scenario you describe could be
> handled in such a way -
>
> You could have a status node in ZooKeeper which every slave will subscribe
> to and update! If one of the slave nodes sees that there have been too many
> connection refused to the Leader by the slaves, the slave could go ahead and
> delete the Leader znode, and force the Leader to give up its leadership. I
> am not describing a deatiled way to do it, but its not very hard to come up
> with a design for this.
>
>
>
> Do you intend to have the Leader and Slaves in different Network (different
> ACLs I mean) protected zones? In that case, it is a legitimate concern else
> I do think assymetric network partition would be very unlikely to happen.
>
> Do you usually see network partitions in such scenarios?
>
> Thanks
> mahadev
>
>
> On 4/30/10 4:05 PM, "Lei Gao" <lg...@linkedin.com> wrote:
>
>> Hi Mahadev,
>>
>> Why would the leader be disconnected from ZK? ZK is fine communicating with
>> the leader in this case. We are talking about asymmetric network failure.
>> Yes. Leader could consider all the slaves being down if it tracks the status
>> of all slaves himself. But I guess if ZK is used for for membership
>> management, neither the leader nor the slaves will be considered
>> disconnected because they can all connect to ZK.
>>
>> Thanks,
>>
>> Lei
>>
>>
>> On 4/30/10 3:47 PM, "Mahadev Konar" <ma...@yahoo-inc.com> wrote:
>>
>>> Hi Lei,
>>>
>>> In this case, the Leader will be disconnected from ZK cluster and will give
>>> up its leadership. Since its disconnected, ZK cluster will realize that the
>>> Leader is dead!....
>>>
>>> When Zk cluster realizes that the Leader is dead (this is because the zk
>>> cluster hasn't heard from the Leader for a certain time.... Configurable via
>>> session timeout parameter), the slaves will be notified of this via watchers
>>> in zookeeper cluster. The slaves will realize that the Leader is gone and
>>> will relect a new Leader and will start working with the new Leader.
>>>
>>> Does that answer your question?
>>>
>>> You might want to look though the documentation of ZK to understand its use
>>> case and how it solves these kind of issues....
>>>
>>> Thanks
>>> mahadev
>>>
>>>
>>> On 4/30/10 2:08 PM, "Lei Gao" <lg...@linkedin.com> wrote:
>>>
>>>> Thank you all for your answers. It clarifies a lot of my confusion about
>>>> the
>>>> service guarantees of ZK. I am still struggling with one failure case (I am
>>>> not trying to be the pain in the neck. But I need to have a full
>>>> understanding of what ZK can offer before I make a decision on whether to
>>>> used it in my cluster.)
>>>>
>>>> Assume the following topology:
>>>>
>>>> Leader ==== ZK cluster
>>>> \\ //
>>>> \\ //
>>>> \\ //
>>>> Slave(s)
>>>>
>>>> If I am asymmetric network failure such that the connection between Leader
>>>> and Slave(s) are broken while all other connections are still alive, would
>>>> my system hang after some point? Because no new leader election will be
>>>> initiated by slaves and the leader can't get the work to slave(s).
>>>>
>>>> Thanks,
>>>>
>>>> Lei
>>>>
>>>> On 4/30/10 1:54 PM, "Ted Dunning" <te...@gmail.com> wrote:
>>>>
>>>>> If one of your user clients can no longer reach one member of the ZK
>>>>> cluster, then it will try to reach another. If it succeeds, then it will
>>>>> continue without any problems as long as the ZK cluster itself is OK.
>>>>>
>>>>> This applies for all the ZK recipes. You will have to be a little bit
>>>>> careful to handle connection loss, but that should get easier soon (and
>>>>> isn't all that difficult anyway).
>>>>>
>>>>> On Fri, Apr 30, 2010 at 1:26 PM, Lei Gao <lg...@linkedin.com> wrote:
>>>>>
>>>>>> I am not talking about the leader election within zookeeper cluster. I
>>>>>> guess
>>>>>> I didn't make the discussion context clear. In my case, I run a cluster
>>>>>> that
>>>>>> uses zookeeper for doing the leader election. Yes, nodes in my cluster
>>>>>> are
>>>>>> the clients of zookeeper. Those nodes depend on zookeeper to elect a new
>>>>>> leader and figure out what the current leader is. So if the zookeeper
>>>>>> (think
>>>>>> of it as a stand-alone entity) becomes unavailabe in the way I've
>>>>>> described
>>>>>> earlier, how can I handle such situation so my cluster can still function
>>>>>> while a majority of nodes still connect to each other (but not to the
>>>>>> zookeeper)?
>>>>>>
>>>>
>>>
>>
>
Re: Question on maintaining leader/membership status in zookeeper
Posted by Patrick Hunt <ph...@apache.org>.
I believe Lei's concern is that the leader and all slaves can talk to
ZK, but the slaves cannot talk to the leader. As a result no work can be
done. However nothing will happen on the ZK side since everyone is
heartbeating properly.
Mahadev I think you came up with a pretty good solution. However since
the leader can see the votes from all the slaves it might just want to
give up the lead itself and "pause" for a while (to give someone else
the chance to be the leader). This would allow the leader to handle the
case better where a single slave cannot talk to the leader, but the rest
of the slaves can communicate fine.
Patrick
On 04/30/2010 04:31 PM, Mahadev Konar wrote:
> Maybe I jumped the gun here but Ted's response to your query is more
> appropriate -
> ------------
>
>
> You can then use ZK in your application to pick a lead machine for other
> operations. In that case, essentially every failure scenario is handled by
> the standard recipe. In your example where the master and slave are cut
> off, but both still have access to ZK, all that will happen is that the
> master cannot communicate with the slave. Both will still be clear about
> who is in which role.
>
> The case where the master is cut off from both ZK and the slave is also
> handled well as is the case where the master is cut off from ZK, but not
> from the slave. In both cases, the master will get a connection loss event
> and stop trying to act like a master and the slave will be notified that the
> master has dropped out of its role.
>
> --------------------------
>
>
>
>
>
> On 4/30/10 4:14 PM, "Mahadev Konar"<ma...@yahoo-inc.com> wrote:
>
>> Hi Lei,
>> Sorry I minsinterpreted your question! The scenario you describe could be
>> handled in such a way -
>>
>> You could have a status node in ZooKeeper which every slave will subscribe
>> to and update! If one of the slave nodes sees that there have been too many
>> connection refused to the Leader by the slaves, the slave could go ahead and
>> delete the Leader znode, and force the Leader to give up its leadership. I
>> am not describing a deatiled way to do it, but its not very hard to come up
>> with a design for this.
>>
>>
>>
>> Do you intend to have the Leader and Slaves in different Network (different
>> ACLs I mean) protected zones? In that case, it is a legitimate concern else
>> I do think assymetric network partition would be very unlikely to happen.
>>
>> Do you usually see network partitions in such scenarios?
>>
>> Thanks
>> mahadev
>>
>>
>> On 4/30/10 4:05 PM, "Lei Gao"<lg...@linkedin.com> wrote:
>>
>>> Hi Mahadev,
>>>
>>> Why would the leader be disconnected from ZK? ZK is fine communicating with
>>> the leader in this case. We are talking about asymmetric network failure.
>>> Yes. Leader could consider all the slaves being down if it tracks the status
>>> of all slaves himself. But I guess if ZK is used for for membership
>>> management, neither the leader nor the slaves will be considered
>>> disconnected because they can all connect to ZK.
>>>
>>> Thanks,
>>>
>>> Lei
>>>
>>>
>>> On 4/30/10 3:47 PM, "Mahadev Konar"<ma...@yahoo-inc.com> wrote:
>>>
>>>> Hi Lei,
>>>>
>>>> In this case, the Leader will be disconnected from ZK cluster and will give
>>>> up its leadership. Since its disconnected, ZK cluster will realize that the
>>>> Leader is dead!....
>>>>
>>>> When Zk cluster realizes that the Leader is dead (this is because the zk
>>>> cluster hasn't heard from the Leader for a certain time.... Configurable via
>>>> session timeout parameter), the slaves will be notified of this via watchers
>>>> in zookeeper cluster. The slaves will realize that the Leader is gone and
>>>> will relect a new Leader and will start working with the new Leader.
>>>>
>>>> Does that answer your question?
>>>>
>>>> You might want to look though the documentation of ZK to understand its use
>>>> case and how it solves these kind of issues....
>>>>
>>>> Thanks
>>>> mahadev
>>>>
>>>>
>>>> On 4/30/10 2:08 PM, "Lei Gao"<lg...@linkedin.com> wrote:
>>>>
>>>>> Thank you all for your answers. It clarifies a lot of my confusion about
>>>>> the
>>>>> service guarantees of ZK. I am still struggling with one failure case (I am
>>>>> not trying to be the pain in the neck. But I need to have a full
>>>>> understanding of what ZK can offer before I make a decision on whether to
>>>>> used it in my cluster.)
>>>>>
>>>>> Assume the following topology:
>>>>>
>>>>> Leader ==== ZK cluster
>>>>> \\ //
>>>>> \\ //
>>>>> \\ //
>>>>> Slave(s)
>>>>>
>>>>> If I am asymmetric network failure such that the connection between Leader
>>>>> and Slave(s) are broken while all other connections are still alive, would
>>>>> my system hang after some point? Because no new leader election will be
>>>>> initiated by slaves and the leader can't get the work to slave(s).
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Lei
>>>>>
>>>>> On 4/30/10 1:54 PM, "Ted Dunning"<te...@gmail.com> wrote:
>>>>>
>>>>>> If one of your user clients can no longer reach one member of the ZK
>>>>>> cluster, then it will try to reach another. If it succeeds, then it will
>>>>>> continue without any problems as long as the ZK cluster itself is OK.
>>>>>>
>>>>>> This applies for all the ZK recipes. You will have to be a little bit
>>>>>> careful to handle connection loss, but that should get easier soon (and
>>>>>> isn't all that difficult anyway).
>>>>>>
>>>>>> On Fri, Apr 30, 2010 at 1:26 PM, Lei Gao<lg...@linkedin.com> wrote:
>>>>>>
>>>>>>> I am not talking about the leader election within zookeeper cluster. I
>>>>>>> guess
>>>>>>> I didn't make the discussion context clear. In my case, I run a cluster
>>>>>>> that
>>>>>>> uses zookeeper for doing the leader election. Yes, nodes in my cluster
>>>>>>> are
>>>>>>> the clients of zookeeper. Those nodes depend on zookeeper to elect a new
>>>>>>> leader and figure out what the current leader is. So if the zookeeper
>>>>>>> (think
>>>>>>> of it as a stand-alone entity) becomes unavailabe in the way I've
>>>>>>> described
>>>>>>> earlier, how can I handle such situation so my cluster can still function
>>>>>>> while a majority of nodes still connect to each other (but not to the
>>>>>>> zookeeper)?
>>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Question on maintaining leader/membership status in zookeeper
Posted by Mahadev Konar <ma...@yahoo-inc.com>.
Maybe I jumped the gun here but Ted's response to your query is more
appropriate -
------------
You can then use ZK in your application to pick a lead machine for other
operations. In that case, essentially every failure scenario is handled by
the standard recipe. In your example where the master and slave are cut
off, but both still have access to ZK, all that will happen is that the
master cannot communicate with the slave. Both will still be clear about
who is in which role.
The case where the master is cut off from both ZK and the slave is also
handled well as is the case where the master is cut off from ZK, but not
from the slave. In both cases, the master will get a connection loss event
and stop trying to act like a master and the slave will be notified that the
master has dropped out of its role.
--------------------------
On 4/30/10 4:14 PM, "Mahadev Konar" <ma...@yahoo-inc.com> wrote:
> Hi Lei,
> Sorry I minsinterpreted your question! The scenario you describe could be
> handled in such a way -
>
> You could have a status node in ZooKeeper which every slave will subscribe
> to and update! If one of the slave nodes sees that there have been too many
> connection refused to the Leader by the slaves, the slave could go ahead and
> delete the Leader znode, and force the Leader to give up its leadership. I
> am not describing a deatiled way to do it, but its not very hard to come up
> with a design for this.
>
>
>
> Do you intend to have the Leader and Slaves in different Network (different
> ACLs I mean) protected zones? In that case, it is a legitimate concern else
> I do think assymetric network partition would be very unlikely to happen.
>
> Do you usually see network partitions in such scenarios?
>
> Thanks
> mahadev
>
>
> On 4/30/10 4:05 PM, "Lei Gao" <lg...@linkedin.com> wrote:
>
>> Hi Mahadev,
>>
>> Why would the leader be disconnected from ZK? ZK is fine communicating with
>> the leader in this case. We are talking about asymmetric network failure.
>> Yes. Leader could consider all the slaves being down if it tracks the status
>> of all slaves himself. But I guess if ZK is used for for membership
>> management, neither the leader nor the slaves will be considered
>> disconnected because they can all connect to ZK.
>>
>> Thanks,
>>
>> Lei
>>
>>
>> On 4/30/10 3:47 PM, "Mahadev Konar" <ma...@yahoo-inc.com> wrote:
>>
>>> Hi Lei,
>>>
>>> In this case, the Leader will be disconnected from ZK cluster and will give
>>> up its leadership. Since its disconnected, ZK cluster will realize that the
>>> Leader is dead!....
>>>
>>> When Zk cluster realizes that the Leader is dead (this is because the zk
>>> cluster hasn't heard from the Leader for a certain time.... Configurable via
>>> session timeout parameter), the slaves will be notified of this via watchers
>>> in zookeeper cluster. The slaves will realize that the Leader is gone and
>>> will relect a new Leader and will start working with the new Leader.
>>>
>>> Does that answer your question?
>>>
>>> You might want to look though the documentation of ZK to understand its use
>>> case and how it solves these kind of issues....
>>>
>>> Thanks
>>> mahadev
>>>
>>>
>>> On 4/30/10 2:08 PM, "Lei Gao" <lg...@linkedin.com> wrote:
>>>
>>>> Thank you all for your answers. It clarifies a lot of my confusion about
>>>> the
>>>> service guarantees of ZK. I am still struggling with one failure case (I am
>>>> not trying to be the pain in the neck. But I need to have a full
>>>> understanding of what ZK can offer before I make a decision on whether to
>>>> used it in my cluster.)
>>>>
>>>> Assume the following topology:
>>>>
>>>> Leader ==== ZK cluster
>>>> \\ //
>>>> \\ //
>>>> \\ //
>>>> Slave(s)
>>>>
>>>> If I am asymmetric network failure such that the connection between Leader
>>>> and Slave(s) are broken while all other connections are still alive, would
>>>> my system hang after some point? Because no new leader election will be
>>>> initiated by slaves and the leader can't get the work to slave(s).
>>>>
>>>> Thanks,
>>>>
>>>> Lei
>>>>
>>>> On 4/30/10 1:54 PM, "Ted Dunning" <te...@gmail.com> wrote:
>>>>
>>>>> If one of your user clients can no longer reach one member of the ZK
>>>>> cluster, then it will try to reach another. If it succeeds, then it will
>>>>> continue without any problems as long as the ZK cluster itself is OK.
>>>>>
>>>>> This applies for all the ZK recipes. You will have to be a little bit
>>>>> careful to handle connection loss, but that should get easier soon (and
>>>>> isn't all that difficult anyway).
>>>>>
>>>>> On Fri, Apr 30, 2010 at 1:26 PM, Lei Gao <lg...@linkedin.com> wrote:
>>>>>
>>>>>> I am not talking about the leader election within zookeeper cluster. I
>>>>>> guess
>>>>>> I didn't make the discussion context clear. In my case, I run a cluster
>>>>>> that
>>>>>> uses zookeeper for doing the leader election. Yes, nodes in my cluster
>>>>>> are
>>>>>> the clients of zookeeper. Those nodes depend on zookeeper to elect a new
>>>>>> leader and figure out what the current leader is. So if the zookeeper
>>>>>> (think
>>>>>> of it as a stand-alone entity) becomes unavailabe in the way I've
>>>>>> described
>>>>>> earlier, how can I handle such situation so my cluster can still function
>>>>>> while a majority of nodes still connect to each other (but not to the
>>>>>> zookeeper)?
>>>>>>
>>>>
>>>
>>
>
Re: Question on maintaining leader/membership status in zookeeper
Posted by Mahadev Konar <ma...@yahoo-inc.com>.
Hi Lei,
Sorry I minsinterpreted your question! The scenario you describe could be
handled in such a way -
You could have a status node in ZooKeeper which every slave will subscribe
to and update! If one of the slave nodes sees that there have been too many
connection refused to the Leader by the slaves, the slave could go ahead and
delete the Leader znode, and force the Leader to give up its leadership. I
am not describing a deatiled way to do it, but its not very hard to come up
with a design for this.
Do you intend to have the Leader and Slaves in different Network (different
ACLs I mean) protected zones? In that case, it is a legitimate concern else
I do think assymetric network partition would be very unlikely to happen.
Do you usually see network partitions in such scenarios?
Thanks
mahadev
On 4/30/10 4:05 PM, "Lei Gao" <lg...@linkedin.com> wrote:
> Hi Mahadev,
>
> Why would the leader be disconnected from ZK? ZK is fine communicating with
> the leader in this case. We are talking about asymmetric network failure.
> Yes. Leader could consider all the slaves being down if it tracks the status
> of all slaves himself. But I guess if ZK is used for for membership
> management, neither the leader nor the slaves will be considered
> disconnected because they can all connect to ZK.
>
> Thanks,
>
> Lei
>
>
> On 4/30/10 3:47 PM, "Mahadev Konar" <ma...@yahoo-inc.com> wrote:
>
>> Hi Lei,
>>
>> In this case, the Leader will be disconnected from ZK cluster and will give
>> up its leadership. Since its disconnected, ZK cluster will realize that the
>> Leader is dead!....
>>
>> When Zk cluster realizes that the Leader is dead (this is because the zk
>> cluster hasn't heard from the Leader for a certain time.... Configurable via
>> session timeout parameter), the slaves will be notified of this via watchers
>> in zookeeper cluster. The slaves will realize that the Leader is gone and
>> will relect a new Leader and will start working with the new Leader.
>>
>> Does that answer your question?
>>
>> You might want to look though the documentation of ZK to understand its use
>> case and how it solves these kind of issues....
>>
>> Thanks
>> mahadev
>>
>>
>> On 4/30/10 2:08 PM, "Lei Gao" <lg...@linkedin.com> wrote:
>>
>>> Thank you all for your answers. It clarifies a lot of my confusion about the
>>> service guarantees of ZK. I am still struggling with one failure case (I am
>>> not trying to be the pain in the neck. But I need to have a full
>>> understanding of what ZK can offer before I make a decision on whether to
>>> used it in my cluster.)
>>>
>>> Assume the following topology:
>>>
>>> Leader ==== ZK cluster
>>> \\ //
>>> \\ //
>>> \\ //
>>> Slave(s)
>>>
>>> If I am asymmetric network failure such that the connection between Leader
>>> and Slave(s) are broken while all other connections are still alive, would
>>> my system hang after some point? Because no new leader election will be
>>> initiated by slaves and the leader can't get the work to slave(s).
>>>
>>> Thanks,
>>>
>>> Lei
>>>
>>> On 4/30/10 1:54 PM, "Ted Dunning" <te...@gmail.com> wrote:
>>>
>>>> If one of your user clients can no longer reach one member of the ZK
>>>> cluster, then it will try to reach another. If it succeeds, then it will
>>>> continue without any problems as long as the ZK cluster itself is OK.
>>>>
>>>> This applies for all the ZK recipes. You will have to be a little bit
>>>> careful to handle connection loss, but that should get easier soon (and
>>>> isn't all that difficult anyway).
>>>>
>>>> On Fri, Apr 30, 2010 at 1:26 PM, Lei Gao <lg...@linkedin.com> wrote:
>>>>
>>>>> I am not talking about the leader election within zookeeper cluster. I
>>>>> guess
>>>>> I didn't make the discussion context clear. In my case, I run a cluster
>>>>> that
>>>>> uses zookeeper for doing the leader election. Yes, nodes in my cluster are
>>>>> the clients of zookeeper. Those nodes depend on zookeeper to elect a new
>>>>> leader and figure out what the current leader is. So if the zookeeper
>>>>> (think
>>>>> of it as a stand-alone entity) becomes unavailabe in the way I've
>>>>> described
>>>>> earlier, how can I handle such situation so my cluster can still function
>>>>> while a majority of nodes still connect to each other (but not to the
>>>>> zookeeper)?
>>>>>
>>>
>>
>
Re: Question on maintaining leader/membership status in zookeeper
Posted by Lei Gao <lg...@linkedin.com>.
Hi Mahadev,
Why would the leader be disconnected from ZK? ZK is fine communicating with
the leader in this case. We are talking about asymmetric network failure.
Yes. Leader could consider all the slaves being down if it tracks the status
of all slaves himself. But I guess if ZK is used for for membership
management, neither the leader nor the slaves will be considered
disconnected because they can all connect to ZK.
Thanks,
Lei
On 4/30/10 3:47 PM, "Mahadev Konar" <ma...@yahoo-inc.com> wrote:
> Hi Lei,
>
> In this case, the Leader will be disconnected from ZK cluster and will give
> up its leadership. Since its disconnected, ZK cluster will realize that the
> Leader is dead!....
>
> When Zk cluster realizes that the Leader is dead (this is because the zk
> cluster hasn't heard from the Leader for a certain time.... Configurable via
> session timeout parameter), the slaves will be notified of this via watchers
> in zookeeper cluster. The slaves will realize that the Leader is gone and
> will relect a new Leader and will start working with the new Leader.
>
> Does that answer your question?
>
> You might want to look though the documentation of ZK to understand its use
> case and how it solves these kind of issues....
>
> Thanks
> mahadev
>
>
> On 4/30/10 2:08 PM, "Lei Gao" <lg...@linkedin.com> wrote:
>
>> Thank you all for your answers. It clarifies a lot of my confusion about the
>> service guarantees of ZK. I am still struggling with one failure case (I am
>> not trying to be the pain in the neck. But I need to have a full
>> understanding of what ZK can offer before I make a decision on whether to
>> used it in my cluster.)
>>
>> Assume the following topology:
>>
>> Leader ==== ZK cluster
>> \\ //
>> \\ //
>> \\ //
>> Slave(s)
>>
>> If I am asymmetric network failure such that the connection between Leader
>> and Slave(s) are broken while all other connections are still alive, would
>> my system hang after some point? Because no new leader election will be
>> initiated by slaves and the leader can't get the work to slave(s).
>>
>> Thanks,
>>
>> Lei
>>
>> On 4/30/10 1:54 PM, "Ted Dunning" <te...@gmail.com> wrote:
>>
>>> If one of your user clients can no longer reach one member of the ZK
>>> cluster, then it will try to reach another. If it succeeds, then it will
>>> continue without any problems as long as the ZK cluster itself is OK.
>>>
>>> This applies for all the ZK recipes. You will have to be a little bit
>>> careful to handle connection loss, but that should get easier soon (and
>>> isn't all that difficult anyway).
>>>
>>> On Fri, Apr 30, 2010 at 1:26 PM, Lei Gao <lg...@linkedin.com> wrote:
>>>
>>>> I am not talking about the leader election within zookeeper cluster. I
>>>> guess
>>>> I didn't make the discussion context clear. In my case, I run a cluster
>>>> that
>>>> uses zookeeper for doing the leader election. Yes, nodes in my cluster are
>>>> the clients of zookeeper. Those nodes depend on zookeeper to elect a new
>>>> leader and figure out what the current leader is. So if the zookeeper
>>>> (think
>>>> of it as a stand-alone entity) becomes unavailabe in the way I've described
>>>> earlier, how can I handle such situation so my cluster can still function
>>>> while a majority of nodes still connect to each other (but not to the
>>>> zookeeper)?
>>>>
>>
>
Re: Question on maintaining leader/membership status in zookeeper
Posted by Mahadev Konar <ma...@yahoo-inc.com>.
Hi Lei,
In this case, the Leader will be disconnected from ZK cluster and will give
up its leadership. Since its disconnected, ZK cluster will realize that the
Leader is dead!....
When Zk cluster realizes that the Leader is dead (this is because the zk
cluster hasn't heard from the Leader for a certain time.... Configurable via
session timeout parameter), the slaves will be notified of this via watchers
in zookeeper cluster. The slaves will realize that the Leader is gone and
will relect a new Leader and will start working with the new Leader.
Does that answer your question?
You might want to look though the documentation of ZK to understand its use
case and how it solves these kind of issues....
Thanks
mahadev
On 4/30/10 2:08 PM, "Lei Gao" <lg...@linkedin.com> wrote:
> Thank you all for your answers. It clarifies a lot of my confusion about the
> service guarantees of ZK. I am still struggling with one failure case (I am
> not trying to be the pain in the neck. But I need to have a full
> understanding of what ZK can offer before I make a decision on whether to
> used it in my cluster.)
>
> Assume the following topology:
>
> Leader ==== ZK cluster
> \\ //
> \\ //
> \\ //
> Slave(s)
>
> If I am asymmetric network failure such that the connection between Leader
> and Slave(s) are broken while all other connections are still alive, would
> my system hang after some point? Because no new leader election will be
> initiated by slaves and the leader can't get the work to slave(s).
>
> Thanks,
>
> Lei
>
> On 4/30/10 1:54 PM, "Ted Dunning" <te...@gmail.com> wrote:
>
>> If one of your user clients can no longer reach one member of the ZK
>> cluster, then it will try to reach another. If it succeeds, then it will
>> continue without any problems as long as the ZK cluster itself is OK.
>>
>> This applies for all the ZK recipes. You will have to be a little bit
>> careful to handle connection loss, but that should get easier soon (and
>> isn't all that difficult anyway).
>>
>> On Fri, Apr 30, 2010 at 1:26 PM, Lei Gao <lg...@linkedin.com> wrote:
>>
>>> I am not talking about the leader election within zookeeper cluster. I
>>> guess
>>> I didn't make the discussion context clear. In my case, I run a cluster
>>> that
>>> uses zookeeper for doing the leader election. Yes, nodes in my cluster are
>>> the clients of zookeeper. Those nodes depend on zookeeper to elect a new
>>> leader and figure out what the current leader is. So if the zookeeper
>>> (think
>>> of it as a stand-alone entity) becomes unavailabe in the way I've described
>>> earlier, how can I handle such situation so my cluster can still function
>>> while a majority of nodes still connect to each other (but not to the
>>> zookeeper)?
>>>
>
Re: Question on maintaining leader/membership status in zookeeper
Posted by Lei Gao <lg...@linkedin.com>.
Thank you all for your answers. It clarifies a lot of my confusion about the
service guarantees of ZK. I am still struggling with one failure case (I am
not trying to be the pain in the neck. But I need to have a full
understanding of what ZK can offer before I make a decision on whether to
used it in my cluster.)
Assume the following topology:
Leader ==== ZK cluster
\\ //
\\ //
\\ //
Slave(s)
If I am asymmetric network failure such that the connection between Leader
and Slave(s) are broken while all other connections are still alive, would
my system hang after some point? Because no new leader election will be
initiated by slaves and the leader can't get the work to slave(s).
Thanks,
Lei
On 4/30/10 1:54 PM, "Ted Dunning" <te...@gmail.com> wrote:
> If one of your user clients can no longer reach one member of the ZK
> cluster, then it will try to reach another. If it succeeds, then it will
> continue without any problems as long as the ZK cluster itself is OK.
>
> This applies for all the ZK recipes. You will have to be a little bit
> careful to handle connection loss, but that should get easier soon (and
> isn't all that difficult anyway).
>
> On Fri, Apr 30, 2010 at 1:26 PM, Lei Gao <lg...@linkedin.com> wrote:
>
>> I am not talking about the leader election within zookeeper cluster. I
>> guess
>> I didn't make the discussion context clear. In my case, I run a cluster
>> that
>> uses zookeeper for doing the leader election. Yes, nodes in my cluster are
>> the clients of zookeeper. Those nodes depend on zookeeper to elect a new
>> leader and figure out what the current leader is. So if the zookeeper
>> (think
>> of it as a stand-alone entity) becomes unavailabe in the way I've described
>> earlier, how can I handle such situation so my cluster can still function
>> while a majority of nodes still connect to each other (but not to the
>> zookeeper)?
>>
Re: Question on maintaining leader/membership status in zookeeper
Posted by Ted Dunning <te...@gmail.com>.
If one of your user clients can no longer reach one member of the ZK
cluster, then it will try to reach another. If it succeeds, then it will
continue without any problems as long as the ZK cluster itself is OK.
This applies for all the ZK recipes. You will have to be a little bit
careful to handle connection loss, but that should get easier soon (and
isn't all that difficult anyway).
On Fri, Apr 30, 2010 at 1:26 PM, Lei Gao <lg...@linkedin.com> wrote:
> I am not talking about the leader election within zookeeper cluster. I
> guess
> I didn't make the discussion context clear. In my case, I run a cluster
> that
> uses zookeeper for doing the leader election. Yes, nodes in my cluster are
> the clients of zookeeper. Those nodes depend on zookeeper to elect a new
> leader and figure out what the current leader is. So if the zookeeper
> (think
> of it as a stand-alone entity) becomes unavailabe in the way I've described
> earlier, how can I handle such situation so my cluster can still function
> while a majority of nodes still connect to each other (but not to the
> zookeeper)?
>
Re: Question on maintaining leader/membership status in zookeeper
Posted by Patrick Hunt <ph...@apache.org>.
It really depends on the type of leader election you have implemented.
You can do simple leader election such as we describe in the recipes, in
that case clients won't notice the leader changed unless they are
connected. You could implement some more complex protocol, but typically
the recipe we have is sufficient.
In my experience what you should do also depends on whether you have an
"active" leader or a passive one.
A passive leader is one that is connected to by other hosts and thereby
performs some action, an active leader is one that initiates actions.
Typically in the passive case if the leader gets disconnected from the
server you just want to wait until you get reconnected to determine if
you are still the leader. (hosts just won't connect to you if zk cluster
notifies them that you are no longer the leader)
However in the active case while you are disconnected from the cluster
someone else may have become the leader, in that case you should stop
initiating actions (typically). You may want to stop acting as the
leader as soon as you become disconnected from the ensemble because you
have no way to know if/when someone else was elected as the leader.
Patrick
On 04/30/2010 01:26 PM, Lei Gao wrote:
> Hi Henry,
>
> I am not talking about the leader election within zookeeper cluster. I guess
> I didn't make the discussion context clear. In my case, I run a cluster that
> uses zookeeper for doing the leader election. Yes, nodes in my cluster are
> the clients of zookeeper. Those nodes depend on zookeeper to elect a new
> leader and figure out what the current leader is. So if the zookeeper (think
> of it as a stand-alone entity) becomes unavailabe in the way I've described
> earlier, how can I handle such situation so my cluster can still function
> while a majority of nodes still connect to each other (but not to the
> zookeeper)?
>
> Thanks,
>
> Lei
>
>
> On 4/30/10 1:10 PM, "Henry Robinson"<he...@cloudera.com> wrote:
>
>> Hi Lei -
>>
>> The 'user cluster' (by which I think you mean the set of clients of
>> ZooKeeper?) plays no part in leader election. If a majority of ZooKeeper
>> server nodes can talk to each other, a new leader can be elected. Clients of
>> the minority server partition will be disconnected - if they too cannot
>> reach the majority partition then they will not be able to reconnect.
>>
>> Hope this helps,
>> Henry
>>
>> On 30 April 2010 12:45, Lei Gao<lg...@linkedin.com> wrote:
>>
>>> Hi Ted,
>>>
>>> I 100% agree with what you said. But my question is more about what if my
>>> zookeeper service cluster is partitioned from a majority of nodes in my USER
>>> CLUSTER. In this case, the majority nodes in one network partition can¹t
>>> select a new leader because zookeeper is out of reach.
>>>
>>> Another example will be that if there is an asymmetric network failure
>>> where a majority of nodes from the USER CLUSTER can¹t reach the leader while
>>> the zookeeper still can. How does zookeeper handle such situation?
>>>
>>> Thanks,
>>>
>>> Lei
>>>
>>> On 4/30/10 12:24 PM, "Ted Dunning"<te...@gmail.com> wrote:
>>>
>>> There are a variety of situations that can trigger a new leader election
>>> and a few that can cause the cluster to be unable to elect a new leader.
>>> Isolation of just the leader is one of the situations that will cause a new
>>> leader election. Isolation of nodes into groups smaller than the quorum
>>> will result in the cluster freezing.
>>>
>>> On Fri, Apr 30, 2010 at 11:56 AM, Lei Gao<lg...@linkedin.com> wrote:
>>> Hi,
>>>
>>> I have a general question on how zookeeper can maintain its view of the
>>> user cluster (that zookeeper manages) that is consistent with the nodes in
>>> the user cluster. In other words, when zookeeper considers the current
>>> leader is unavailable, does it really guarantee that a majority of nodes in
>>> the user cluster can¹t reach the current leader? The same question applies
>>> to the membership service as well. Because the zookeeper can be partitioned
>>> from a majority of the nodes in the user cluster. How does the zookeeper
>>> handle situations like this?
>>>
>>> Thanks,
>>>
>>> Lei
>>>
>>>
>>>
>>
>
Re: Question on maintaining leader/membership status in zookeeper
Posted by Mahadev Konar <ma...@yahoo-inc.com>.
Hi Lei,
In this case, its up to application to decide what to do when this happens.
The application will be notified that its disconnected from the ZooKeeper
cluster. In such a case some of the applications might decide to not proceed
at all, (since it might lead to some state corruption) and some others might
decide on using cached values, wherein stale values are fine for correctness
of the system. Its up to you to decide what you would want to do in such a
situation.
Also, usually you would want to set up ZooKeeper clusters in such a way that
this should not be possible... Like across switches....
In this case, the application will be able to access one of the zookeeper
servers on the zookeeper cluster and it will be highly unlikely that they
arent able to reach any one of those.
Hope this helps.
Thanks
mahadev
On 4/30/10 1:26 PM, "Lei Gao" <lg...@linkedin.com> wrote:
> Hi Henry,
>
> I am not talking about the leader election within zookeeper cluster. I guess
> I didn't make the discussion context clear. In my case, I run a cluster that
> uses zookeeper for doing the leader election. Yes, nodes in my cluster are
> the clients of zookeeper. Those nodes depend on zookeeper to elect a new
> leader and figure out what the current leader is. So if the zookeeper (think
> of it as a stand-alone entity) becomes unavailabe in the way I've described
> earlier, how can I handle such situation so my cluster can still function
> while a majority of nodes still connect to each other (but not to the
> zookeeper)?
>
> Thanks,
>
> Lei
>
>
> On 4/30/10 1:10 PM, "Henry Robinson" <he...@cloudera.com> wrote:
>
>> Hi Lei -
>>
>> The 'user cluster' (by which I think you mean the set of clients of
>> ZooKeeper?) plays no part in leader election. If a majority of ZooKeeper
>> server nodes can talk to each other, a new leader can be elected. Clients of
>> the minority server partition will be disconnected - if they too cannot
>> reach the majority partition then they will not be able to reconnect.
>>
>> Hope this helps,
>> Henry
>>
>> On 30 April 2010 12:45, Lei Gao <lg...@linkedin.com> wrote:
>>
>>> Hi Ted,
>>>
>>> I 100% agree with what you said. But my question is more about what if my
>>> zookeeper service cluster is partitioned from a majority of nodes in my USER
>>> CLUSTER. In this case, the majority nodes in one network partition can¹t
>>> select a new leader because zookeeper is out of reach.
>>>
>>> Another example will be that if there is an asymmetric network failure
>>> where a majority of nodes from the USER CLUSTER can¹t reach the leader while
>>> the zookeeper still can. How does zookeeper handle such situation?
>>>
>>> Thanks,
>>>
>>> Lei
>>>
>>> On 4/30/10 12:24 PM, "Ted Dunning" <te...@gmail.com> wrote:
>>>
>>> There are a variety of situations that can trigger a new leader election
>>> and a few that can cause the cluster to be unable to elect a new leader.
>>> Isolation of just the leader is one of the situations that will cause a new
>>> leader election. Isolation of nodes into groups smaller than the quorum
>>> will result in the cluster freezing.
>>>
>>> On Fri, Apr 30, 2010 at 11:56 AM, Lei Gao <lg...@linkedin.com> wrote:
>>> Hi,
>>>
>>> I have a general question on how zookeeper can maintain its view of the
>>> user cluster (that zookeeper manages) that is consistent with the nodes in
>>> the user cluster. In other words, when zookeeper considers the current
>>> leader is unavailable, does it really guarantee that a majority of nodes in
>>> the user cluster can¹t reach the current leader? The same question applies
>>> to the membership service as well. Because the zookeeper can be partitioned
>>> from a majority of the nodes in the user cluster. How does the zookeeper
>>> handle situations like this?
>>>
>>> Thanks,
>>>
>>> Lei
>>>
>>>
>>>
>>
>
Re: Question on maintaining leader/membership status in zookeeper
Posted by Lei Gao <lg...@linkedin.com>.
Hi Henry,
I am not talking about the leader election within zookeeper cluster. I guess
I didn't make the discussion context clear. In my case, I run a cluster that
uses zookeeper for doing the leader election. Yes, nodes in my cluster are
the clients of zookeeper. Those nodes depend on zookeeper to elect a new
leader and figure out what the current leader is. So if the zookeeper (think
of it as a stand-alone entity) becomes unavailabe in the way I've described
earlier, how can I handle such situation so my cluster can still function
while a majority of nodes still connect to each other (but not to the
zookeeper)?
Thanks,
Lei
On 4/30/10 1:10 PM, "Henry Robinson" <he...@cloudera.com> wrote:
> Hi Lei -
>
> The 'user cluster' (by which I think you mean the set of clients of
> ZooKeeper?) plays no part in leader election. If a majority of ZooKeeper
> server nodes can talk to each other, a new leader can be elected. Clients of
> the minority server partition will be disconnected - if they too cannot
> reach the majority partition then they will not be able to reconnect.
>
> Hope this helps,
> Henry
>
> On 30 April 2010 12:45, Lei Gao <lg...@linkedin.com> wrote:
>
>> Hi Ted,
>>
>> I 100% agree with what you said. But my question is more about what if my
>> zookeeper service cluster is partitioned from a majority of nodes in my USER
>> CLUSTER. In this case, the majority nodes in one network partition can¹t
>> select a new leader because zookeeper is out of reach.
>>
>> Another example will be that if there is an asymmetric network failure
>> where a majority of nodes from the USER CLUSTER can¹t reach the leader while
>> the zookeeper still can. How does zookeeper handle such situation?
>>
>> Thanks,
>>
>> Lei
>>
>> On 4/30/10 12:24 PM, "Ted Dunning" <te...@gmail.com> wrote:
>>
>> There are a variety of situations that can trigger a new leader election
>> and a few that can cause the cluster to be unable to elect a new leader.
>> Isolation of just the leader is one of the situations that will cause a new
>> leader election. Isolation of nodes into groups smaller than the quorum
>> will result in the cluster freezing.
>>
>> On Fri, Apr 30, 2010 at 11:56 AM, Lei Gao <lg...@linkedin.com> wrote:
>> Hi,
>>
>> I have a general question on how zookeeper can maintain its view of the
>> user cluster (that zookeeper manages) that is consistent with the nodes in
>> the user cluster. In other words, when zookeeper considers the current
>> leader is unavailable, does it really guarantee that a majority of nodes in
>> the user cluster can¹t reach the current leader? The same question applies
>> to the membership service as well. Because the zookeeper can be partitioned
>> from a majority of the nodes in the user cluster. How does the zookeeper
>> handle situations like this?
>>
>> Thanks,
>>
>> Lei
>>
>>
>>
>
Re: Question on maintaining leader/membership status in zookeeper
Posted by Henry Robinson <he...@cloudera.com>.
Hi Lei -
The 'user cluster' (by which I think you mean the set of clients of
ZooKeeper?) plays no part in leader election. If a majority of ZooKeeper
server nodes can talk to each other, a new leader can be elected. Clients of
the minority server partition will be disconnected - if they too cannot
reach the majority partition then they will not be able to reconnect.
Hope this helps,
Henry
On 30 April 2010 12:45, Lei Gao <lg...@linkedin.com> wrote:
> Hi Ted,
>
> I 100% agree with what you said. But my question is more about what if my
> zookeeper service cluster is partitioned from a majority of nodes in my USER
> CLUSTER. In this case, the majority nodes in one network partition can’t
> select a new leader because zookeeper is out of reach.
>
> Another example will be that if there is an asymmetric network failure
> where a majority of nodes from the USER CLUSTER can’t reach the leader while
> the zookeeper still can. How does zookeeper handle such situation?
>
> Thanks,
>
> Lei
>
> On 4/30/10 12:24 PM, "Ted Dunning" <te...@gmail.com> wrote:
>
> There are a variety of situations that can trigger a new leader election
> and a few that can cause the cluster to be unable to elect a new leader.
> Isolation of just the leader is one of the situations that will cause a new
> leader election. Isolation of nodes into groups smaller than the quorum
> will result in the cluster freezing.
>
> On Fri, Apr 30, 2010 at 11:56 AM, Lei Gao <lg...@linkedin.com> wrote:
> Hi,
>
> I have a general question on how zookeeper can maintain its view of the
> user cluster (that zookeeper manages) that is consistent with the nodes in
> the user cluster. In other words, when zookeeper considers the current
> leader is unavailable, does it really guarantee that a majority of nodes in
> the user cluster can’t reach the current leader? The same question applies
> to the membership service as well. Because the zookeeper can be partitioned
> from a majority of the nodes in the user cluster. How does the zookeeper
> handle situations like this?
>
> Thanks,
>
> Lei
>
>
>
--
Henry Robinson
Software Engineer
Cloudera
415-994-6679
Re: Question on maintaining leader/membership status in zookeeper
Posted by Mahadev Konar <ma...@yahoo-inc.com>.
Hi Lei,
I think you might be confusing Leader Election within ZooKeeper itself with
electing a leader for your application? Is that so?
Leader Election within ZooKeeper is totally internal to zookeeper service
and is not visible to applications.
I am little confused, what your problem statement is? Can you please explain
it from the view point of your application?
Thanks
mahadev
On 4/30/10 12:45 PM, "Lei Gao" <lg...@linkedin.com> wrote:
> Hi Ted,
>
> I 100% agree with what you said. But my question is more about what if my
> zookeeper service cluster is partitioned from a majority of nodes in my USER
> CLUSTER. In this case, the majority nodes in one network partition can¹t
> select a new leader because zookeeper is out of reach.
>
> Another example will be that if there is an asymmetric network failure where a
> majority of nodes from the USER CLUSTER can¹t reach the leader while the
> zookeeper still can. How does zookeeper handle such situation?
>
> Thanks,
>
> Lei
>
> On 4/30/10 12:24 PM, "Ted Dunning" <te...@gmail.com> wrote:
>
> There are a variety of situations that can trigger a new leader election and a
> few that can cause the cluster to be unable to elect a new leader. Isolation
> of just the leader is one of the situations that will cause a new leader
> election. Isolation of nodes into groups smaller than the quorum will result
> in the cluster freezing.
>
> On Fri, Apr 30, 2010 at 11:56 AM, Lei Gao <lg...@linkedin.com> wrote:
> Hi,
>
> I have a general question on how zookeeper can maintain its view of the user
> cluster (that zookeeper manages) that is consistent with the nodes in the user
> cluster. In other words, when zookeeper considers the current leader is
> unavailable, does it really guarantee that a majority of nodes in the user
> cluster can¹t reach the current leader? The same question applies to the
> membership service as well. Because the zookeeper can be partitioned from a
> majority of the nodes in the user cluster. How does the zookeeper handle
> situations like this?
>
> Thanks,
>
> Lei
>
>
Re: Question on maintaining leader/membership status in zookeeper
Posted by Lei Gao <lg...@linkedin.com>.
Hi Ted,
I 100% agree with what you said. But my question is more about what if my zookeeper service cluster is partitioned from a majority of nodes in my USER CLUSTER. In this case, the majority nodes in one network partition can’t select a new leader because zookeeper is out of reach.
Another example will be that if there is an asymmetric network failure where a majority of nodes from the USER CLUSTER can’t reach the leader while the zookeeper still can. How does zookeeper handle such situation?
Thanks,
Lei
On 4/30/10 12:24 PM, "Ted Dunning" <te...@gmail.com> wrote:
There are a variety of situations that can trigger a new leader election and a few that can cause the cluster to be unable to elect a new leader. Isolation of just the leader is one of the situations that will cause a new leader election. Isolation of nodes into groups smaller than the quorum will result in the cluster freezing.
On Fri, Apr 30, 2010 at 11:56 AM, Lei Gao <lg...@linkedin.com> wrote:
Hi,
I have a general question on how zookeeper can maintain its view of the user cluster (that zookeeper manages) that is consistent with the nodes in the user cluster. In other words, when zookeeper considers the current leader is unavailable, does it really guarantee that a majority of nodes in the user cluster can’t reach the current leader? The same question applies to the membership service as well. Because the zookeeper can be partitioned from a majority of the nodes in the user cluster. How does the zookeeper handle situations like this?
Thanks,
Lei
Re: Question on maintaining leader/membership status in zookeeper
Posted by Ted Dunning <te...@gmail.com>.
There are a variety of situations that can trigger a new leader election and
a few that can cause the cluster to be unable to elect a new leader.
Isolation of just the leader is one of the situations that will cause a new
leader election. Isolation of nodes into groups smaller than the quorum
will result in the cluster freezing.
On Fri, Apr 30, 2010 at 11:56 AM, Lei Gao <lg...@linkedin.com> wrote:
> Hi,
>
> I have a general question on how zookeeper can maintain its view of the
> user cluster (that zookeeper manages) that is consistent with the nodes in
> the user cluster. In other words, when zookeeper considers the current
> leader is unavailable, does it really guarantee that a majority of nodes in
> the user cluster can’t reach the current leader? The same question applies
> to the membership service as well. Because the zookeeper can be partitioned
> from a majority of the nodes in the user cluster. How does the zookeeper
> handle situations like this?
>
> Thanks,
>
> Lei
>