You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@curator.apache.org by Ricardo Ferreira <ri...@gmail.com> on 2015/01/27 10:20:07 UTC

Regarding leadership status

Hello all,


I'm using Zookeeper (through Curator) for leadership election on an
application I'm currently developing.

Some operations can only be written by one of the members of the cluster,
so I'm using a node's leadership
status to this effect.

For example, let's say a message arrives to all nodes but only one should
do something about it. What I do
is to check if the node is leader (through LeaderSelector#hasLeadership()),
and if it is then it does whatever
it needs to do with the message. If it's not, it simply ignores it.

My question is this: How can I be sure a node *really is* the leader? What
if the leader node is actually down but
this hasn't been detected by Zookeeper yet? Then the non-leader nodes will
ignore the message and it will
never be processed.

How is this usually dealt with?


Regards

Re: Regarding leadership status

Posted by Ricardo Ferreira <ri...@gmail.com>.

Cam, that's not exactly the problem I'm facing.

My question is that all nodes that are connected to Zookeeper think they're
not leaders, because the node that was a leader died and ZK didn't caught
the event yet.

In my case this will result in some operations to never be executed,
because their the leader's responsibility.

On Tue, Jan 27, 2015 at 11:04 AM, Cameron McKenzie <ca...@apache.org>
wrote:

> hey Ricardo,
> That's an interesting problem.
>
> If a node is disconnected from ZK it will report that it did not have
> leadership, regardless of whether its ephemeral node on ZK has been removed
> or not.
>
> If a node is not connected to ZK then I guess it could persist messages
> until it is reconnected and then process them afterwards. The issue being
> that it may no longer be leader in which case the messages would not be
> processed by any node. Not sure if it's possible / practical, but I guess
> the node could re-broadcast all of the persisted messages to all nodes (or
> just the current leader) once it reconnected to ZK.
>
> cheers
> Cam
>
> On Tue, Jan 27, 2015 at 8:20 PM, Ricardo Ferreira <ri...@gmail.com>
> wrote:
>
>> Hello all,
>>
>>
>> I'm using Zookeeper (through Curator) for leadership election on an
>> application I'm currently developing.
>>
>> Some operations can only be written by one of the members of the cluster,
>> so I'm using a node's leadership
>> status to this effect.
>>
>> For example, let's say a message arrives to all nodes but only one should
>> do something about it. What I do
>> is to check if the node is leader (through
>> LeaderSelector#hasLeadership()), and if it is then it does whatever
>> it needs to do with the message. If it's not, it simply ignores it.
>>
>> My question is this: How can I be sure a node *really is* the leader?
>> What if the leader node is actually down but
>> this hasn't been detected by Zookeeper yet? Then the non-leader nodes
>> will ignore the message and it will
>> never be processed.
>>
>> How is this usually dealt with?
>>
>>
>> Regards
>>
>
>

Re: Regarding leadership status

Posted by Cameron McKenzie <ca...@apache.org>.

hey Ricardo,
That's an interesting problem.

If a node is disconnected from ZK it will report that it did not have
leadership, regardless of whether its ephemeral node on ZK has been removed
or not.

If a node is not connected to ZK then I guess it could persist messages
until it is reconnected and then process them afterwards. The issue being
that it may no longer be leader in which case the messages would not be
processed by any node. Not sure if it's possible / practical, but I guess
the node could re-broadcast all of the persisted messages to all nodes (or
just the current leader) once it reconnected to ZK.

cheers
Cam

On Tue, Jan 27, 2015 at 8:20 PM, Ricardo Ferreira <ri...@gmail.com>
wrote:

> Hello all,
>
>
> I'm using Zookeeper (through Curator) for leadership election on an
> application I'm currently developing.
>
> Some operations can only be written by one of the members of the cluster,
> so I'm using a node's leadership
> status to this effect.
>
> For example, let's say a message arrives to all nodes but only one should
> do something about it. What I do
> is to check if the node is leader (through
> LeaderSelector#hasLeadership()), and if it is then it does whatever
> it needs to do with the message. If it's not, it simply ignores it.
>
> My question is this: How can I be sure a node *really is* the leader?
> What if the leader node is actually down but
> this hasn't been detected by Zookeeper yet? Then the non-leader nodes will
> ignore the message and it will
> never be processed.
>
> How is this usually dealt with?
>
>
> Regards
>

Re: Regarding leadership status

Posted by Jordan Zimmerman <jo...@jordanzimmerman.com>.

Can you give a more detailed example?

In ZooKeeper, a server writes a unique value to the ZK cluster and, if successful, proceeds to do its work. This is the majority case. Curator’s leader recipes implement this. Are you saying, that you elect a leader but another instance does the work? Why would you need this?

-JZ

On January 27, 2015 at 2:10:57 PM, Ricardo Ferreira (ricardojsfer@gmail.com) wrote:

I was using leadership status to restrict the node who acts upon an action request. Actions don't necessarily originate from a leader node.
If the leader goes down actions are still requested... But the way I'm modeling it and from what you said, it seems they'll be lost.

On Tue, Jan 27, 2015 at 7:05 PM, Jordan Zimmerman <jo...@jordanzimmerman.com> wrote:
The actions will be performed by the leader node. So, if the leader goes down no actions occur. So, I don’t follow your concern.

On January 27, 2015 at 2:04:03 PM, Ricardo Ferreira (ricardojsfer@gmail.com) wrote:

Sure, I understand that.

There are also no guarantees if the leader node goes down and an action must be performed in between heartbeats The Zookeeper client wont yet be aware a node is down and the cluster's state will be inconsistent, right?

On Tue, Jan 27, 2015 at 5:54 PM, Jordan Zimmerman <jo...@jordanzimmerman.com> wrote:
No. If a non-leader dies there is not a new leader election. The current leader stays leader.

-JZ

On January 27, 2015 at 12:50:43 PM, Ricardo Ferreira (ricardojsfer@gmail.com) wrote:

And from what you described, it gets worse: If a non-leader dies, leadership election will take place and a new leader might be elected, right?

Re: Regarding leadership status

Posted by Manikumar Reddy <ku...@nmsworks.co.in>.

If I understand  correctly , your question  is how to avoid message loss
during leadership changes.

This is related to architecture  of your application.  You can persist your
messages to systems like Kafka, activemq, rabbitmq  etc and leader can
process them. During leadership changes you won't lose any data.
On Jan 28, 2015 12:41 AM, "Ricardo Ferreira" <ri...@gmail.com> wrote:

> I was using leadership status to restrict the node who acts upon an action
> request. Actions don't necessarily originate from a leader node.
> If the leader goes down actions are still requested... But the way I'm
> modeling it and from what you said, it seems they'll be lost.
>
> On Tue, Jan 27, 2015 at 7:05 PM, Jordan Zimmerman <
> jordan@jordanzimmerman.com> wrote:
>
>> The actions will be performed by the leader node. So, if the leader goes
>> down no actions occur. So, I don’t follow your concern.
>>
>>
>> On January 27, 2015 at 2:04:03 PM, Ricardo Ferreira (
>> ricardojsfer@gmail.com) wrote:
>>
>> Sure, I understand that.
>>
>> There are also no guarantees if the leader node goes down and an action
>> must be performed in between heartbeats The Zookeeper client wont yet be
>> aware a node is down and the cluster's state will be inconsistent, right?
>>
>> On Tue, Jan 27, 2015 at 5:54 PM, Jordan Zimmerman <
>> jordan@jordanzimmerman.com> wrote:
>>
>>>  No. If a non-leader dies there is not a new leader election. The
>>> current leader stays leader.
>>>
>>>  -JZ
>>>
>>> On January 27, 2015 at 12:50:43 PM, Ricardo Ferreira (
>>> ricardojsfer@gmail.com) wrote:
>>>
>>>  And from what you described, it gets worse: If a non-leader dies,
>>> leadership election will take place and a new leader might be elected,
>>> right?
>>>
>>>
>>
>

Re: Regarding leadership status

Posted by Ricardo Ferreira <ri...@gmail.com>.

I was using leadership status to restrict the node who acts upon an action
request. Actions don't necessarily originate from a leader node.
If the leader goes down actions are still requested... But the way I'm
modeling it and from what you said, it seems they'll be lost.

On Tue, Jan 27, 2015 at 7:05 PM, Jordan Zimmerman <
jordan@jordanzimmerman.com> wrote:

> The actions will be performed by the leader node. So, if the leader goes
> down no actions occur. So, I don’t follow your concern.
>
>
> On January 27, 2015 at 2:04:03 PM, Ricardo Ferreira (
> ricardojsfer@gmail.com) wrote:
>
> Sure, I understand that.
>
> There are also no guarantees if the leader node goes down and an action
> must be performed in between heartbeats The Zookeeper client wont yet be
> aware a node is down and the cluster's state will be inconsistent, right?
>
> On Tue, Jan 27, 2015 at 5:54 PM, Jordan Zimmerman <
> jordan@jordanzimmerman.com> wrote:
>
>>  No. If a non-leader dies there is not a new leader election. The
>> current leader stays leader.
>>
>>  -JZ
>>
>> On January 27, 2015 at 12:50:43 PM, Ricardo Ferreira (
>> ricardojsfer@gmail.com) wrote:
>>
>>  And from what you described, it gets worse: If a non-leader dies,
>> leadership election will take place and a new leader might be elected,
>> right?
>>
>>
>

Re: Regarding leadership status

Posted by Jordan Zimmerman <jo...@jordanzimmerman.com>.

The actions will be performed by the leader node. So, if the leader goes down no actions occur. So, I don’t follow your concern. 


On January 27, 2015 at 2:04:03 PM, Ricardo Ferreira (ricardojsfer@gmail.com) wrote:

Sure, I understand that.

There are also no guarantees if the leader node goes down and an action must be performed in between heartbeats The Zookeeper client wont yet be aware a node is down and the cluster's state will be inconsistent, right?

On Tue, Jan 27, 2015 at 5:54 PM, Jordan Zimmerman <jo...@jordanzimmerman.com> wrote:
No. If a non-leader dies there is not a new leader election. The current leader stays leader.

-JZ

On January 27, 2015 at 12:50:43 PM, Ricardo Ferreira (ricardojsfer@gmail.com) wrote:

And from what you described, it gets worse: If a non-leader dies, leadership election will take place and a new leader might be elected, right?

Re: Regarding leadership status

Posted by Ricardo Ferreira <ri...@gmail.com>.

Sure, I understand that.

There are also no guarantees if the leader node goes down and an action
must be performed in between heartbeats The Zookeeper client wont yet be
aware a node is down and the cluster's state will be inconsistent, right?

On Tue, Jan 27, 2015 at 5:54 PM, Jordan Zimmerman <
jordan@jordanzimmerman.com> wrote:

> No. If a non-leader dies there is not a new leader election. The current
> leader stays leader.
>
> -JZ
>
> On January 27, 2015 at 12:50:43 PM, Ricardo Ferreira (
> ricardojsfer@gmail.com) wrote:
>
> And from what you described, it gets worse: If a non-leader dies,
> leadership election will take place and a new leader might be elected,
> right?
>
>

Re: Regarding leadership status

Posted by Jordan Zimmerman <jo...@jordanzimmerman.com>.

No. If a non-leader dies there is not a new leader election. The current leader stays leader.

-JZ

On January 27, 2015 at 12:50:43 PM, Ricardo Ferreira (ricardojsfer@gmail.com) wrote:

And from what you described, it gets worse: If a non-leader dies, leadership election will take place and a new leader might be elected, right?

Re: Regarding leadership status

Posted by Jordan Zimmerman <jo...@jordanzimmerman.com>.

This is CAP. In a distributed system, each node must be able to deal with network partitions. If you are partitioned, you cannot know the status of the cluster. Therefore, your code must be able to deal with a situation where it’s not possible to know leaders, locks, etc.

If an action which is the leader's responsibility to execute (e.g. write to DB) must be performed during that time it will be lost.
If you can’t determine the leader, your application must go into a suspended state until the partition is repaired. There is no way around this.

-JZ

On January 27, 2015 at 12:50:43 PM, Ricardo Ferreira (ricardojsfer@gmail.com) wrote:

Jordan,

That is exactly the problem I'm facing: at a given moment you can be leader-less.

If an action which is the leader's responsibility to execute (e.g. write to DB) must be performed during that time it will be lost.

And from what you described, it gets worse: If a non-leader dies, leadership election will take place and a new leader might be elected, right?

On Tue, Jan 27, 2015 at 5:23 PM, Jordan Zimmerman <jo...@jordanzimmerman.com> wrote:
The process of leadership loss due to network issues is this:

* The current leader's ZooKeeper client will try to send its heartbeat
* The send will fail due to network failure
* The ZK client will pass KeeperState.Disconnected and cause Curator to go into SUSPENDED mode. Any users of leader selectors, etc. should see this and consider themselves no longer leader
* Because the heartbeat is 2/3 of a session, there will be no other leader at this point
* After session expiration, the ZK server will delete the ephemeral node and cause the process of leader selection, etc.

So, as you can see, if you properly monitor the connection state you never have to worry about proper leadership.

-Jordan

On January 27, 2015 at 4:21:40 AM, Ricardo Ferreira (ricardojsfer@gmail.com) wrote:

Hello all,

I'm using Zookeeper (through Curator) for leadership election on an application I'm currently developing.

Some operations can only be written by one of the members of the cluster, so I'm using a node's leadership
status to this effect.

For example, let's say a message arrives to all nodes but only one should do something about it. What I do
is to check if the node is leader (through LeaderSelector#hasLeadership()), and if it is then it does whatever
it needs to do with the message. If it's not, it simply ignores it.

My question is this: How can I be sure a node really is the leader? What if the leader node is actually down but
this hasn't been detected by Zookeeper yet? Then the non-leader nodes will ignore the message and it will
never be processed.

How is this usually dealt with?

Regards

Re: Regarding leadership status

Posted by Ricardo Ferreira <ri...@gmail.com>.

Jordan,

That is exactly the problem I'm facing: at a given moment you can be
leader-less.

If an action which is the leader's responsibility to execute (e.g. write to
DB) must be performed during that time it will be lost.

And from what you described, it gets worse: If a non-leader dies,
leadership election will take place and a new leader might be elected,
right?

On Tue, Jan 27, 2015 at 5:23 PM, Jordan Zimmerman <
jordan@jordanzimmerman.com> wrote:

> The process of leadership loss due to network issues is this:
>
> * The current leader's ZooKeeper client will try to send its heartbeat
> * The send will fail due to network failure
> * The ZK client will pass KeeperState.Disconnected and cause Curator to go
> into SUSPENDED mode. Any users of leader selectors, etc. should see this
> and consider themselves no longer leader
> * Because the heartbeat is 2/3 of a session, there will be no other leader
> at this point
> * After session expiration, the ZK server will delete the ephemeral node
> and cause the process of leader selection, etc.
>
> So, as you can see, if you properly monitor the connection state you never
> have to worry about proper leadership.
>
> -Jordan
>
>
> On January 27, 2015 at 4:21:40 AM, Ricardo Ferreira (
> ricardojsfer@gmail.com) wrote:
>
> Hello all,
>
>
> I'm using Zookeeper (through Curator) for leadership election on an
> application I'm currently developing.
>
> Some operations can only be written by one of the members of the cluster,
> so I'm using a node's leadership
> status to this effect.
>
> For example, let's say a message arrives to all nodes but only one should
> do something about it. What I do
> is to check if the node is leader (through
> LeaderSelector#hasLeadership()), and if it is then it does whatever
> it needs to do with the message. If it's not, it simply ignores it.
>
> My question is this: How can I be sure a node *really is* the leader?
> What if the leader node is actually down but
> this hasn't been detected by Zookeeper yet? Then the non-leader nodes will
> ignore the message and it will
> never be processed.
>
> How is this usually dealt with?
>
>
> Regards
>
>

Re: Regarding leadership status

Posted by Jordan Zimmerman <jo...@jordanzimmerman.com>.

The process of leadership loss due to network issues is this:

So, as you can see, if you properly monitor the connection state you never have to worry about proper leadership.

-Jordan

On January 27, 2015 at 4:21:40 AM, Ricardo Ferreira (ricardojsfer@gmail.com) wrote:

Hello all,

I'm using Zookeeper (through Curator) for leadership election on an application I'm currently developing.

Some operations can only be written by one of the members of the cluster, so I'm using a node's leadership
status to this effect.

How is this usually dealt with?

Regards