You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by Jérémie BORDIER <je...@gmail.com> on 2011/11/14 00:40:02 UTC
Missing session state handling in most Leader Election implementations
Hi Folks,
We have been playing around with ZooKeeper for a few weeks now, and
reading carefully through the documentation I noticed this statement:
If you are using watches, you must look for the connected watch event.
When a ZooKeeper client disconnects from a server, you will not
receive notification of changes until reconnected. If you are watching
for a znode to come into existence, you will miss the event if the
znode is created and deleted while you are disconnected.
As noticed in ZOOKEEPER-1209, this can cause really important issues.
As Leader election is one of the most demanded feature / recipe, I
would really like to see the official recipe fixed and fully
functional.
I decided to throw a look at other implementations of the leader
election and surprisingly, none of them seemed to care about the
Disconnected / Expired / SyncConnected events in a simple way. Here's
my quick analysis of what they do, and I'd love to know whether I'm
missing something or if they are really wrong:
Twitter commons library Election recipe is based on their "Group"
implementation, with EPHEMERAL|SEQUENTIAL nodes in the same way of the
official LES algorithm. Looking at the Group impl (
https://github.com/twitter/commons/blob/master/src/java/com/twitter/common/zookeeper/Group.java
), they handle the Expired event and retry to join / watch, which
makes a getClient() that will recreate the connection if the
connection has expired. This looks fine for the Expired event, but
what about Disconnected / SyncConnected events ? Nothing.
Netflix' curator library has an approach where the leader acquires an
inter process mutex, backed by a group with EPHEMERAL|SEQUENTIAL nodes
also. Netflix's library has a big advantage: It has a built in API for
retrying actions, so leader election will try to acquire the lock, and
retry if anything goes wrong in the middle. In case of any event, the
loop waiting for the lock will be notified, and will retry in case of
any failure, so a Disconnected or Expired event would be handled
properly. On the other side, it seems that once the leader has been
elected, the event just seems to be ignored. This may lead to the same
split brain issue than the original LES example. (see
https://github.com/Netflix/curator/blob/master/curator-recipes/src/main/java/com/netflix/curator/framework/recipes/locks/LockInternals.java
for details).
Here's all I came up to so far. If you guys have the time to throw a
look to these implementations, I would love to know if I missed
something. So, I think this split brain issue may almost never happen,
but as usually what should never happen hits you hard when you don't
expect it. A robust Leader election implementation would be really
great to have.
Cheers,
Jérémie
Re: Missing session state handling in most Leader Election implementations
Posted by Jérémie BORDIER <je...@gmail.com>.
Just a quick post to point at that the leader election example that
was posted on the list earlier today is very clean and handle the
disconnected / expired cases.
https://github.com/cyberroadie/zookeeper-leader/
Jérémie
On Fri, Nov 18, 2011 at 7:04 PM, Jordan Zimmerman
<jz...@netflix.com> wrote:
> I just did a quickie test. If the cluster goes down you get the Disconnect
> but do not get a session expiration. So, there wouldn't be an opportunity
> to transition from SUSPENDED to LOST (unless the client makes another ZK
> call). So, this brings me back to doing the background sync().
>
> -JZ
>
> On 11/18/11 9:52 AM, "Ted Dunning" <te...@gmail.com> wrote:
>
>>Is the background sync even necessary? The ZK client itself will
>>re-establish connection if it can.
>>
>>I think that LOST should only be sent on session expiration.
>>
>>On Fri, Nov 18, 2011 at 1:07 AM, Jordan Zimmerman
>><jz...@netflix.com>wrote:
>>
>>> FYI
>>>
>>> Curator now has a staged connection notification mechanism for dealing
>>> with issues like this. When the Curator managed connection receives a
>>> Disconnect, it posts a message to listeners that the connection is
>>> SUSPENDED. If the connection can be re-established (via a background
>>>sync()
>>> using the current retry policy) the listeners receive RECONNECTED
>>>otherwise
>>> they receive LOST. Thus, users of the Curator LeaderSelector can know if
>>> they should pause their leader activity and/or stop leader activity.
>>>
>>> -JZ
>>> ________________________________________
>>> From: Ted Dunning [ted.dunning@gmail.com]
>>> Sent: Monday, November 14, 2011 6:24 PM
>>> To: user@zookeeper.apache.org
>>> Subject: Re: Missing session state handling in most Leader Election
>>> implementations
>>>
>>> On Mon, Nov 14, 2011 at 2:41 PM, Jordan Zimmerman
>>><jzimmerman@netflix.com
>>> >wrote:
>>>
>>> > It turns out that this is tricky to solve. When the server you're
>>> > connected to goes down, you get a
>>>Watcher.Event.KeeperState.Disconnected.
>>> > However, it could be that you are able to reconnect to another server
>>>so
>>> > the disconnected event should be ignored.
>>>
>>>
>>> The event should not be ignored. The master should pause in being a
>>> master, but not unload any major data structures. If it reconnects
>>> instantly, then it should continue as if nothing had happened. You can
>>> also have a time limit for how long you wait before you decide to pause
>>> operation as master. As you increase that time, you increase the
>>> probability of two masters existing at the same time. If the reconnect
>>> happens before the timeout, you don't need to both the master.
>>>
>
>
--
Jérémie 'ahFeel' BORDIER
Re: Missing session state handling in most Leader Election
implementations
Posted by Jordan Zimmerman <jz...@netflix.com>.
The sync is necessary to allow the configured retryPolicy to be applied.
However, you're right that I could just wait for the next connection or
the session expiration. I'll give that some thought.
-JZ
On 11/18/11 9:52 AM, "Ted Dunning" <te...@gmail.com> wrote:
>Is the background sync even necessary? The ZK client itself will
>re-establish connection if it can.
>
>I think that LOST should only be sent on session expiration.
>
>On Fri, Nov 18, 2011 at 1:07 AM, Jordan Zimmerman
><jz...@netflix.com>wrote:
>
>> FYI
>>
>> Curator now has a staged connection notification mechanism for dealing
>> with issues like this. When the Curator managed connection receives a
>> Disconnect, it posts a message to listeners that the connection is
>> SUSPENDED. If the connection can be re-established (via a background
>>sync()
>> using the current retry policy) the listeners receive RECONNECTED
>>otherwise
>> they receive LOST. Thus, users of the Curator LeaderSelector can know if
>> they should pause their leader activity and/or stop leader activity.
>>
>> -JZ
>> ________________________________________
>> From: Ted Dunning [ted.dunning@gmail.com]
>> Sent: Monday, November 14, 2011 6:24 PM
>> To: user@zookeeper.apache.org
>> Subject: Re: Missing session state handling in most Leader Election
>> implementations
>>
>> On Mon, Nov 14, 2011 at 2:41 PM, Jordan Zimmerman
>><jzimmerman@netflix.com
>> >wrote:
>>
>> > It turns out that this is tricky to solve. When the server you're
>> > connected to goes down, you get a
>>Watcher.Event.KeeperState.Disconnected.
>> > However, it could be that you are able to reconnect to another server
>>so
>> > the disconnected event should be ignored.
>>
>>
>> The event should not be ignored. The master should pause in being a
>> master, but not unload any major data structures. If it reconnects
>> instantly, then it should continue as if nothing had happened. You can
>> also have a time limit for how long you wait before you decide to pause
>> operation as master. As you increase that time, you increase the
>> probability of two masters existing at the same time. If the reconnect
>> happens before the timeout, you don't need to both the master.
>>
Re: Missing session state handling in most Leader Election
implementations
Posted by Jordan Zimmerman <jz...@netflix.com>.
I just did a quickie test. If the cluster goes down you get the Disconnect
but do not get a session expiration. So, there wouldn't be an opportunity
to transition from SUSPENDED to LOST (unless the client makes another ZK
call). So, this brings me back to doing the background sync().
-JZ
On 11/18/11 9:52 AM, "Ted Dunning" <te...@gmail.com> wrote:
>Is the background sync even necessary? The ZK client itself will
>re-establish connection if it can.
>
>I think that LOST should only be sent on session expiration.
>
>On Fri, Nov 18, 2011 at 1:07 AM, Jordan Zimmerman
><jz...@netflix.com>wrote:
>
>> FYI
>>
>> Curator now has a staged connection notification mechanism for dealing
>> with issues like this. When the Curator managed connection receives a
>> Disconnect, it posts a message to listeners that the connection is
>> SUSPENDED. If the connection can be re-established (via a background
>>sync()
>> using the current retry policy) the listeners receive RECONNECTED
>>otherwise
>> they receive LOST. Thus, users of the Curator LeaderSelector can know if
>> they should pause their leader activity and/or stop leader activity.
>>
>> -JZ
>> ________________________________________
>> From: Ted Dunning [ted.dunning@gmail.com]
>> Sent: Monday, November 14, 2011 6:24 PM
>> To: user@zookeeper.apache.org
>> Subject: Re: Missing session state handling in most Leader Election
>> implementations
>>
>> On Mon, Nov 14, 2011 at 2:41 PM, Jordan Zimmerman
>><jzimmerman@netflix.com
>> >wrote:
>>
>> > It turns out that this is tricky to solve. When the server you're
>> > connected to goes down, you get a
>>Watcher.Event.KeeperState.Disconnected.
>> > However, it could be that you are able to reconnect to another server
>>so
>> > the disconnected event should be ignored.
>>
>>
>> The event should not be ignored. The master should pause in being a
>> master, but not unload any major data structures. If it reconnects
>> instantly, then it should continue as if nothing had happened. You can
>> also have a time limit for how long you wait before you decide to pause
>> operation as master. As you increase that time, you increase the
>> probability of two masters existing at the same time. If the reconnect
>> happens before the timeout, you don't need to both the master.
>>
Re: Missing session state handling in most Leader Election implementations
Posted by Ted Dunning <te...@gmail.com>.
Is the background sync even necessary? The ZK client itself will
re-establish connection if it can.
I think that LOST should only be sent on session expiration.
On Fri, Nov 18, 2011 at 1:07 AM, Jordan Zimmerman <jz...@netflix.com>wrote:
> FYI
>
> Curator now has a staged connection notification mechanism for dealing
> with issues like this. When the Curator managed connection receives a
> Disconnect, it posts a message to listeners that the connection is
> SUSPENDED. If the connection can be re-established (via a background sync()
> using the current retry policy) the listeners receive RECONNECTED otherwise
> they receive LOST. Thus, users of the Curator LeaderSelector can know if
> they should pause their leader activity and/or stop leader activity.
>
> -JZ
> ________________________________________
> From: Ted Dunning [ted.dunning@gmail.com]
> Sent: Monday, November 14, 2011 6:24 PM
> To: user@zookeeper.apache.org
> Subject: Re: Missing session state handling in most Leader Election
> implementations
>
> On Mon, Nov 14, 2011 at 2:41 PM, Jordan Zimmerman <jzimmerman@netflix.com
> >wrote:
>
> > It turns out that this is tricky to solve. When the server you're
> > connected to goes down, you get a Watcher.Event.KeeperState.Disconnected.
> > However, it could be that you are able to reconnect to another server so
> > the disconnected event should be ignored.
>
>
> The event should not be ignored. The master should pause in being a
> master, but not unload any major data structures. If it reconnects
> instantly, then it should continue as if nothing had happened. You can
> also have a time limit for how long you wait before you decide to pause
> operation as master. As you increase that time, you increase the
> probability of two masters existing at the same time. If the reconnect
> happens before the timeout, you don't need to both the master.
>
RE: Missing session state handling in most Leader Election
implementations
Posted by Jordan Zimmerman <jz...@netflix.com>.
FYI
Curator now has a staged connection notification mechanism for dealing with issues like this. When the Curator managed connection receives a Disconnect, it posts a message to listeners that the connection is SUSPENDED. If the connection can be re-established (via a background sync() using the current retry policy) the listeners receive RECONNECTED otherwise they receive LOST. Thus, users of the Curator LeaderSelector can know if they should pause their leader activity and/or stop leader activity.
-JZ
________________________________________
From: Ted Dunning [ted.dunning@gmail.com]
Sent: Monday, November 14, 2011 6:24 PM
To: user@zookeeper.apache.org
Subject: Re: Missing session state handling in most Leader Election implementations
On Mon, Nov 14, 2011 at 2:41 PM, Jordan Zimmerman <jz...@netflix.com>wrote:
> It turns out that this is tricky to solve. When the server you're
> connected to goes down, you get a Watcher.Event.KeeperState.Disconnected.
> However, it could be that you are able to reconnect to another server so
> the disconnected event should be ignored.
The event should not be ignored. The master should pause in being a
master, but not unload any major data structures. If it reconnects
instantly, then it should continue as if nothing had happened. You can
also have a time limit for how long you wait before you decide to pause
operation as master. As you increase that time, you increase the
probability of two masters existing at the same time. If the reconnect
happens before the timeout, you don't need to both the master.
Re: Missing session state handling in most Leader Election implementations
Posted by Ted Dunning <te...@gmail.com>.
On Mon, Nov 14, 2011 at 2:41 PM, Jordan Zimmerman <jz...@netflix.com>wrote:
> It turns out that this is tricky to solve. When the server you're
> connected to goes down, you get a Watcher.Event.KeeperState.Disconnected.
> However, it could be that you are able to reconnect to another server so
> the disconnected event should be ignored.
The event should not be ignored. The master should pause in being a
master, but not unload any major data structures. If it reconnects
instantly, then it should continue as if nothing had happened. You can
also have a time limit for how long you wait before you decide to pause
operation as master. As you increase that time, you increase the
probability of two masters existing at the same time. If the reconnect
happens before the timeout, you don't need to both the master.
> My solution is to watch for
> Watcher.Event.KeeperState.Disconnected and then execute a sync() (using
> the currently configured retry policy). If that sync fails, Curator will
> call the unhandledError() method of the LeaderSelectorListener. This seems
> like the best way to handle this. Thoughts?
>
I don't like this as well as pausing.
>
> As an aside, as part of working on this I now have a TestingCluster class
> that will create, in memory, n ZooKeeper servers in an ensemble. This
> could be useful to everyone :)
>
Very.
Can you inject events into this cluster? That is what I have missed.
Re: Missing session state handling in most Leader Election implementations
Posted by Jérémie BORDIER <je...@gmail.com>.
Jordan, thanks for fixing this so quickly. You can now count me in the
happy curator users ;-)
As for the official LES example, it would be great if a more
experienced dev could throw a look to the patch.
Jérémie
On Tue, Nov 15, 2011 at 12:13 AM, Jordan Zimmerman
<jz...@netflix.com> wrote:
> The default retry policy is good (exponential back-off).
>
>>Maven central seems to be quite behind the tags on your github.
> I'm trying to keep it up to date. However, it takes about a day for the
> various caches on the net to pick it up. I'll release a new version today
> with this change.
>
> -JZ
>
>
>
> On 11/14/11 3:08 PM, "Jérémie BORDIER" <je...@gmail.com> wrote:
>
>>Hello Jordan,
>>
>>This sounds like a good way to go, assuming the retry policy is
>>reasonable enough. By the way, what RetryPolicy would you recommend ?
>>Also, what's completely off topic but what's the best way to grab
>>latest curator releases ? Maven central seems to be quite behind the
>>tags on your github.
>>
>>Thanks again !
>>
>>Jérémie
>>
>>On Mon, Nov 14, 2011 at 11:41 PM, Jordan Zimmerman
>><jz...@netflix.com> wrote:
>>> It turns out that this is tricky to solve. When the server you're
>>> connected to goes down, you get a
>>>Watcher.Event.KeeperState.Disconnected.
>>> However, it could be that you are able to reconnect to another server so
>>> the disconnected event should be ignored. My solution is to watch for
>>> Watcher.Event.KeeperState.Disconnected and then execute a sync() (using
>>> the currently configured retry policy). If that sync fails, Curator will
>>> call the unhandledError() method of the LeaderSelectorListener. This
>>>seems
>>> like the best way to handle this. Thoughts?
>>>
>>> As an aside, as part of working on this I now have a TestingCluster
>>>class
>>> that will create, in memory, n ZooKeeper servers in an ensemble. This
>>> could be useful to everyone :)
>>>
>>> -JZ
>>>
>>> On 11/14/11 3:25 AM, "Jérémie BORDIER" <je...@gmail.com>
>>>wrote:
>>>
>>>>I'd be really happy to
>>>>contribute as we plan on using Curator, but I think this may impact
>>>>all the recipes so you're probably the best person to link these bits
>>>>together.
>>>
>>>
>>
>>
>>
>>--
>>Jérémie 'ahFeel' BORDIER
>>
>
>
--
Jérémie 'ahFeel' BORDIER
Re: Missing session state handling in most Leader Election
implementations
Posted by Jordan Zimmerman <jz...@netflix.com>.
The default retry policy is good (exponential back-off).
>Maven central seems to be quite behind the tags on your github.
I'm trying to keep it up to date. However, it takes about a day for the
various caches on the net to pick it up. I'll release a new version today
with this change.
-JZ
On 11/14/11 3:08 PM, "Jérémie BORDIER" <je...@gmail.com> wrote:
>Hello Jordan,
>
>This sounds like a good way to go, assuming the retry policy is
>reasonable enough. By the way, what RetryPolicy would you recommend ?
>Also, what's completely off topic but what's the best way to grab
>latest curator releases ? Maven central seems to be quite behind the
>tags on your github.
>
>Thanks again !
>
>Jérémie
>
>On Mon, Nov 14, 2011 at 11:41 PM, Jordan Zimmerman
><jz...@netflix.com> wrote:
>> It turns out that this is tricky to solve. When the server you're
>> connected to goes down, you get a
>>Watcher.Event.KeeperState.Disconnected.
>> However, it could be that you are able to reconnect to another server so
>> the disconnected event should be ignored. My solution is to watch for
>> Watcher.Event.KeeperState.Disconnected and then execute a sync() (using
>> the currently configured retry policy). If that sync fails, Curator will
>> call the unhandledError() method of the LeaderSelectorListener. This
>>seems
>> like the best way to handle this. Thoughts?
>>
>> As an aside, as part of working on this I now have a TestingCluster
>>class
>> that will create, in memory, n ZooKeeper servers in an ensemble. This
>> could be useful to everyone :)
>>
>> -JZ
>>
>> On 11/14/11 3:25 AM, "Jérémie BORDIER" <je...@gmail.com>
>>wrote:
>>
>>>I'd be really happy to
>>>contribute as we plan on using Curator, but I think this may impact
>>>all the recipes so you're probably the best person to link these bits
>>>together.
>>
>>
>
>
>
>--
>Jérémie 'ahFeel' BORDIER
>
Re: Missing session state handling in most Leader Election implementations
Posted by Jérémie BORDIER <je...@gmail.com>.
Hello Jordan,
This sounds like a good way to go, assuming the retry policy is
reasonable enough. By the way, what RetryPolicy would you recommend ?
Also, what's completely off topic but what's the best way to grab
latest curator releases ? Maven central seems to be quite behind the
tags on your github.
Thanks again !
Jérémie
On Mon, Nov 14, 2011 at 11:41 PM, Jordan Zimmerman
<jz...@netflix.com> wrote:
> It turns out that this is tricky to solve. When the server you're
> connected to goes down, you get a Watcher.Event.KeeperState.Disconnected.
> However, it could be that you are able to reconnect to another server so
> the disconnected event should be ignored. My solution is to watch for
> Watcher.Event.KeeperState.Disconnected and then execute a sync() (using
> the currently configured retry policy). If that sync fails, Curator will
> call the unhandledError() method of the LeaderSelectorListener. This seems
> like the best way to handle this. Thoughts?
>
> As an aside, as part of working on this I now have a TestingCluster class
> that will create, in memory, n ZooKeeper servers in an ensemble. This
> could be useful to everyone :)
>
> -JZ
>
> On 11/14/11 3:25 AM, "Jérémie BORDIER" <je...@gmail.com> wrote:
>
>>I'd be really happy to
>>contribute as we plan on using Curator, but I think this may impact
>>all the recipes so you're probably the best person to link these bits
>>together.
>
>
--
Jérémie 'ahFeel' BORDIER
Re: Missing session state handling in most Leader Election
implementations
Posted by Jordan Zimmerman <jz...@netflix.com>.
It turns out that this is tricky to solve. When the server you're
connected to goes down, you get a Watcher.Event.KeeperState.Disconnected.
However, it could be that you are able to reconnect to another server so
the disconnected event should be ignored. My solution is to watch for
Watcher.Event.KeeperState.Disconnected and then execute a sync() (using
the currently configured retry policy). If that sync fails, Curator will
call the unhandledError() method of the LeaderSelectorListener. This seems
like the best way to handle this. Thoughts?
As an aside, as part of working on this I now have a TestingCluster class
that will create, in memory, n ZooKeeper servers in an ensemble. This
could be useful to everyone :)
-JZ
On 11/14/11 3:25 AM, "Jérémie BORDIER" <je...@gmail.com> wrote:
>I'd be really happy to
>contribute as we plan on using Curator, but I think this may impact
>all the recipes so you're probably the best person to link these bits
>together.
Re: Missing session state handling in most Leader Election implementations
Posted by Jérémie BORDIER <je...@gmail.com>.
@Patrick: Well, there seems to already be a JIRA (see my original
message) opened for this particular issue on the LES recipe, but the
provided patch looks really way too complex to me. It makes the recipe
really unreadable, while the leader election principle with ephemeral
nodes is fairly simple.
Thanks anyway for your kindness, If I have time I'll give a shot at
providing another patch :-)
@Ted: Thanks for your suggestion. I'm a little bit concerned about
setting very aggressive session timeout because our scenario is cross
data center.
Jérémie
On Mon, Nov 14, 2011 at 7:00 PM, Patrick Hunt <ph...@apache.org> wrote:
> On Mon, Nov 14, 2011 at 3:25 AM, Jérémie BORDIER
> <je...@gmail.com> wrote:
>> Good to hear. I was really wondering if I was missing something but
>> took the risk to sound like an idiot on the list, really happy to have
>> pointed that out.
>
> This stuff is hard, we try to answer any question respectfully
> regardless of type. Feel free to ask questions at any time, this one
> is esp interesting and something we should more clearly document.
> Thanks!
>
> It would be great if you could create a jira for this, esp great if
> you could provide a patch to the docs or an update to the FAQ, recipes
> (doc or implementation), etc...
> https://cwiki.apache.org/confluence/display/ZOOKEEPER/HowToContribute
>
> Regards,
>
> Patrick
>
--
Jérémie 'ahFeel' BORDIER
Re: Missing session state handling in most Leader Election implementations
Posted by Patrick Hunt <ph...@apache.org>.
On Mon, Nov 14, 2011 at 3:25 AM, Jérémie BORDIER
<je...@gmail.com> wrote:
> Good to hear. I was really wondering if I was missing something but
> took the risk to sound like an idiot on the list, really happy to have
> pointed that out.
This stuff is hard, we try to answer any question respectfully
regardless of type. Feel free to ask questions at any time, this one
is esp interesting and something we should more clearly document.
Thanks!
It would be great if you could create a jira for this, esp great if
you could provide a patch to the docs or an update to the FAQ, recipes
(doc or implementation), etc...
https://cwiki.apache.org/confluence/display/ZOOKEEPER/HowToContribute
Regards,
Patrick
Re: Missing session state handling in most Leader Election implementations
Posted by Ted Dunning <te...@gmail.com>.
You should be good to go with a very aggressive session timeout. The lack
of correct handling for disconnects should not hurt you and may actually
help since there may be clients who can access the master who is
partitioned away from the ZK cluster, but not access the newly elected
master.
On Mon, Nov 14, 2011 at 3:25 AM, Jérémie BORDIER
<je...@gmail.com>wrote:
> Ted, can't add much to what you said, totally agree. We already
> planned on having a watchdog on this anyway. Having several master for
> a very short time wouldn't be a problem for us, but this must be an
> ephemeral state. (We use master election to determine a single host
> responsible of generating unique 64 bit IDs with the first 16 bits
> dedicated to an incremental "master id", so when a new master gets
> elected, the whole range changes. Having 2 masters for a few seconds
> wouldn't be an issue as the IDs cannot overlap, but still that's
> something we want to avoid as much as possible).
>
Re: Missing session state handling in most Leader Election implementations
Posted by Jérémie BORDIER <je...@gmail.com>.
Hello Guys,
Good to hear. I was really wondering if I was missing something but
took the risk to sound like an idiot on the list, really happy to have
pointed that out.
Jordan, as you said, pinging CuratorZookeeperClient.isConnected would
be a way to do this, but after a quick look at what isConnected does
and how the underlying state does take care of Disconnected / Expired
events, I really think it would be a shame to have another loop
pinging instead of linking the code properly so the events are
propagated up to the Election listener. I'd be really happy to
contribute as we plan on using Curator, but I think this may impact
all the recipes so you're probably the best person to link these bits
together.
Ted, can't add much to what you said, totally agree. We already
planned on having a watchdog on this anyway. Having several master for
a very short time wouldn't be a problem for us, but this must be an
ephemeral state. (We use master election to determine a single host
responsible of generating unique 64 bit IDs with the first 16 bits
dedicated to an incremental "master id", so when a new master gets
elected, the whole range changes. Having 2 masters for a few seconds
wouldn't be an issue as the IDs cannot overlap, but still that's
something we want to avoid as much as possible).
Jérémie
On Mon, Nov 14, 2011 at 7:01 AM, Ted Dunning <te...@gmail.com> wrote:
> In general all leader elections have to pay attention to disconnection.
> The action they need to take when disconnection occurs varies according to
> the application.
>
> Concisely put, the application needs to specify whether it wants (a) at
> most one leader, but possibly zero, (b) at least one leader, but possibly
> more than one or (c) as close to one leader, but possibly more than one or
> zero for (hopefully) short periods of time. I don't think that it is
> possible to do better than (c) in view of PAC considerations, especially in
> the real world where clocks are not even reliably monotonic, but you can
> definitely weight the odds so that actual behavior is much more like (a) or
> (b).
>
> For specification (a), a master must go into a waiting state when
> disconnected and immediately stop acting as master because it cannot know
> precisely whether or when somebody else might be elected leader. Upon
> reconnection, either it will still be master or a session expiration event
> will be received because it is no longer master. There is no way to know
> whether another node has become master in the meantime, nor is there any
> way for other nodes to know that the erstwhile master is still thinking
> that it might still be master (since it may have perished rather than
> simply partitioned away). Because there is a short period between being
> disconnected and recognizing that fact, especially if clocks are confused,
> there is a small possibility of more than one master at any given time, but
> this is very unlikely.
>
> For specification (b), a master should continue acting as master until
> notified by a session expiration that it is not longer such. This has the
> obvious issue that two masters may exist even after a network partition is
> healed. It also has the problem that it takes time for an ephemeral to
> vanish which means that there can be a time with no masters.
>
> For specification (c), a master that is disconnected should try to remain a
> master as long as it thinks that its ephemeral still exists. Such an
> estimation clearly depends on clock accuracy and is subject to uncertainty
> since the actual partition time cannot be known precisely.
>
> I don't know of any open source leader election algorithms that deal with
> these differing needs well. Most don't even deal with strategy (a) very
> well. Neither Netflix library nor Curator seem to at any rate.
>
> It is also unfortunate to only see leader election algorithms that depend
> on sequential ephemerals. The herd effect is not an issue for typical
> leader elections which almost always involve at most a few dozen potential
> leaders and typically involve only a handfull of potential leaders.
>
> In my own implementations, I typically also try to account for the distinct
> possibility that my reasoning about the correctness of my implementation is
> faulty. Typically, I do this by including a watchdog thread that inspects
> the state of the world at intervals to detect when something has gone awry.
> For leader election, this is simply ... if I can get a valid indication
> that there is no leader file, I know that I should immediately start
> another election. Such a watchdog would have avoided the sort of problem
> that was mentioned in the original posting.
>
> In general, I make enough errors that regardless of how certain I am of my
> logic, I have a hard time estimating the probability that I am actually
> correct at higher than 99%. If I want a system that operates correctly at
> a probability of, say, 5 nines, I need to decrease the possibility that I
> have erred by 3 orders of magnitude. One way to do this is to have
> somebody who appears more reliable than I to work on the code. Even so,
> however, I have a hard time believing in my conservative and doubting heart
> of hearts that Camille or Ben are more than ten times more reliable than I
> am which leaves two nines left to go. Likewise, code reviews and unit
> tests are subject to the salesmanship problem and thus cannot give me three
> nines of comfort. A watchdog is nice that way since it has an independent
> probability of being correct that I can imagine would decrease the
> remaining chance of failure by my original 99%.
>
> On Sun, Nov 13, 2011 at 4:56 PM, Jordan Zimmerman <jz...@netflix.com>wrote:
>
>>
>> On 11/13/11 4:45 PM, "Jérémie BORDIER" <je...@gmail.com> wrote:
>> >Hello Jordan,
>> >
>> >Thanks a lot for your answer. I tried to figure out where the handling
>> >of Disconnected / Expired takes place, but so far I understood that to
>> >have notifyClientClosing() called from the Lock, an exception needs to
>> >be raised from somewhere. LockInternal may throw a
>>
>> I think I spoke too soon! You are right that an exception would need to
>> get thrown. Curator will notice the disconnection but the client
>> application would have to do some kind of ZK operation in order to find
>> out about it. This is a good find.
>>
>> I guess I could punt and say that, like checking for an interrupted
>> thread, it's the user's responsibility to periodically check
>> CuratorZookeeperClient.isConnected(). But this might be too much to ask
>> considering my goal with Curator is to alleviate users of doing these
>> kinds of things. I need to think more about thisŠ
>>
>> -JZ
>>
>>
>
--
Jérémie 'ahFeel' BORDIER
Re: Missing session state handling in most Leader Election implementations
Posted by Ted Dunning <te...@gmail.com>.
In general all leader elections have to pay attention to disconnection.
The action they need to take when disconnection occurs varies according to
the application.
Concisely put, the application needs to specify whether it wants (a) at
most one leader, but possibly zero, (b) at least one leader, but possibly
more than one or (c) as close to one leader, but possibly more than one or
zero for (hopefully) short periods of time. I don't think that it is
possible to do better than (c) in view of PAC considerations, especially in
the real world where clocks are not even reliably monotonic, but you can
definitely weight the odds so that actual behavior is much more like (a) or
(b).
For specification (a), a master must go into a waiting state when
disconnected and immediately stop acting as master because it cannot know
precisely whether or when somebody else might be elected leader. Upon
reconnection, either it will still be master or a session expiration event
will be received because it is no longer master. There is no way to know
whether another node has become master in the meantime, nor is there any
way for other nodes to know that the erstwhile master is still thinking
that it might still be master (since it may have perished rather than
simply partitioned away). Because there is a short period between being
disconnected and recognizing that fact, especially if clocks are confused,
there is a small possibility of more than one master at any given time, but
this is very unlikely.
For specification (b), a master should continue acting as master until
notified by a session expiration that it is not longer such. This has the
obvious issue that two masters may exist even after a network partition is
healed. It also has the problem that it takes time for an ephemeral to
vanish which means that there can be a time with no masters.
For specification (c), a master that is disconnected should try to remain a
master as long as it thinks that its ephemeral still exists. Such an
estimation clearly depends on clock accuracy and is subject to uncertainty
since the actual partition time cannot be known precisely.
I don't know of any open source leader election algorithms that deal with
these differing needs well. Most don't even deal with strategy (a) very
well. Neither Netflix library nor Curator seem to at any rate.
It is also unfortunate to only see leader election algorithms that depend
on sequential ephemerals. The herd effect is not an issue for typical
leader elections which almost always involve at most a few dozen potential
leaders and typically involve only a handfull of potential leaders.
In my own implementations, I typically also try to account for the distinct
possibility that my reasoning about the correctness of my implementation is
faulty. Typically, I do this by including a watchdog thread that inspects
the state of the world at intervals to detect when something has gone awry.
For leader election, this is simply ... if I can get a valid indication
that there is no leader file, I know that I should immediately start
another election. Such a watchdog would have avoided the sort of problem
that was mentioned in the original posting.
In general, I make enough errors that regardless of how certain I am of my
logic, I have a hard time estimating the probability that I am actually
correct at higher than 99%. If I want a system that operates correctly at
a probability of, say, 5 nines, I need to decrease the possibility that I
have erred by 3 orders of magnitude. One way to do this is to have
somebody who appears more reliable than I to work on the code. Even so,
however, I have a hard time believing in my conservative and doubting heart
of hearts that Camille or Ben are more than ten times more reliable than I
am which leaves two nines left to go. Likewise, code reviews and unit
tests are subject to the salesmanship problem and thus cannot give me three
nines of comfort. A watchdog is nice that way since it has an independent
probability of being correct that I can imagine would decrease the
remaining chance of failure by my original 99%.
On Sun, Nov 13, 2011 at 4:56 PM, Jordan Zimmerman <jz...@netflix.com>wrote:
>
> On 11/13/11 4:45 PM, "Jérémie BORDIER" <je...@gmail.com> wrote:
> >Hello Jordan,
> >
> >Thanks a lot for your answer. I tried to figure out where the handling
> >of Disconnected / Expired takes place, but so far I understood that to
> >have notifyClientClosing() called from the Lock, an exception needs to
> >be raised from somewhere. LockInternal may throw a
>
> I think I spoke too soon! You are right that an exception would need to
> get thrown. Curator will notice the disconnection but the client
> application would have to do some kind of ZK operation in order to find
> out about it. This is a good find.
>
> I guess I could punt and say that, like checking for an interrupted
> thread, it's the user's responsibility to periodically check
> CuratorZookeeperClient.isConnected(). But this might be too much to ask
> considering my goal with Curator is to alleviate users of doing these
> kinds of things. I need to think more about thisŠ
>
> -JZ
>
>
Re: Missing session state handling in most Leader Election
implementations
Posted by Jordan Zimmerman <jz...@netflix.com>.
On 11/13/11 4:45 PM, "Jérémie BORDIER" <je...@gmail.com> wrote:
>Hello Jordan,
>
>Thanks a lot for your answer. I tried to figure out where the handling
>of Disconnected / Expired takes place, but so far I understood that to
>have notifyClientClosing() called from the Lock, an exception needs to
>be raised from somewhere. LockInternal may throw a
I think I spoke too soon! You are right that an exception would need to
get thrown. Curator will notice the disconnection but the client
application would have to do some kind of ZK operation in order to find
out about it. This is a good find.
I guess I could punt and say that, like checking for an interrupted
thread, it's the user's responsibility to periodically check
CuratorZookeeperClient.isConnected(). But this might be too much to ask
considering my goal with Curator is to alleviate users of doing these
kinds of things. I need to think more about thisŠ
-JZ
Re: Missing session state handling in most Leader Election implementations
Posted by Jérémie BORDIER <je...@gmail.com>.
On Mon, Nov 14, 2011 at 1:28 AM, Jordan Zimmerman
<jz...@netflix.com> wrote:
> On 11/13/11 3:40 PM, "Jérémie BORDIER" <je...@gmail.com> wrote:
>
>>As noticed in ZOOKEEPER-1209, this can cause really important issues.
>>As Leader election is one of the most demanded feature / recipe, I
>>would really like to see the official recipe fixed and fully
>>functional.
>
> Curator handles this in its API. Our LeaderSelector takes a listener
> instance that has two methods:
> public void takeLeadership(CuratorFramework client) throws
> Exception;
>
> public void notifyClientClosing(CuratorFramework client);
>
> The second method is the important one here. You can think of this like
> the JDK's thread interruption mechanism. If a split brain occurs, the
> leader connected to the bad half will have its notifyClientClosing method
> called.
>
>
> BTW - all of our recipes have this kind of mechanism.
>
> -JZ
>
>
Hello Jordan,
Thanks a lot for your answer. I tried to figure out where the handling
of Disconnected / Expired takes place, but so far I understood that to
have notifyClientClosing() called from the Lock, an exception needs to
be raised from somewhere. LockInternal may throw a
ConnectionLossException if we don't find ourselves in the node members
during the lock acquiring process, but I can't find where the
exception comes from when the lock is already acquired (so outside of
the acquiring loop). Could you point me to the right bits ? I believe
what you're saying, just want to fully understand the different parts
before starting using it ;)
Thanks again !
Jérémie
Re: Missing session state handling in most Leader Election
implementations
Posted by Jordan Zimmerman <jz...@netflix.com>.
On 11/13/11 3:40 PM, "Jérémie BORDIER" <je...@gmail.com> wrote:
>As noticed in ZOOKEEPER-1209, this can cause really important issues.
>As Leader election is one of the most demanded feature / recipe, I
>would really like to see the official recipe fixed and fully
>functional.
Curator handles this in its API. Our LeaderSelector takes a listener
instance that has two methods:
public void takeLeadership(CuratorFramework client) throws
Exception;
public void notifyClientClosing(CuratorFramework client);
The second method is the important one here. You can think of this like
the JDK's thread interruption mechanism. If a split brain occurs, the
leader connected to the bad half will have its notifyClientClosing method
called.
BTW - all of our recipes have this kind of mechanism.
-JZ