Disqualify a node from leader election

[Owen Kim <oh...@gmail.com>]

I have a 5-node cluster and have a node that I want to participate in
leader election for quorum but never be a leader candidate. Is it possible
to configure the cluster to never nominate it?

Re: Disqualify a node from leader election

[Kuba Lekstan <ku...@gmail.com>]

I have not seen a server with weight=0 being a leader.

Kuba.


2013/11/25 FPJ <fp...@yahoo.com>

> The problem with this solution is that setting the weight of E to zero will
> make its acknowledgments not count for committing txns or anything in the
> system that requires a quorum. Also, I don't think it prevents E from being
> elected, E is still eligible because it is a participant (but with vote
> zero).
>
> -Flavio
>
> > -----Original Message-----
> > From: Kuba Lekstan [mailto:kuebzky@gmail.com]
> > Sent: 25 November 2013 15:55
> > To: user@zookeeper.apache.org
> > Subject: Re: Disqualify a node from leader election
> >
> > If I understand you correctly as you want the specific node never become
> > the leader.
> >
> > The best solution would be to configure your cluster into 2 groups, put
> first 4
> > servers into group 1 with weight 10 and 5th server into group 2 with
> weight 0.
> > This way 5th servers never gets elected as a leader.
> >
> > http://zookeeper.apache.org/doc/trunk/zookeeperHierarchicalQuorums.ht
> > ml
> >
> >
> > 2013/11/25 FPJ <fp...@yahoo.com>
> >
> > > D and E can only become a leader if they can form a quorum with B
> > > and/or C (assuming you've taken A down). In the case A, D, E are the
> > > only 3 servers able to talk to each other, killing A is going to make
> > > you lose quorum, and if you bring A back up, it might be elected
> > > again, so you might end up in this cycle until the net partition is
> healed or B
> > and/or C are up.
> > >
> > > I suppose we could have an option that tells servers to ignore a given
> > > server when electing a leader, but it is not entirely trivial because
> > > A in the example Ben gave might be the only available server that has
> > > the latest committed txn. We would need a mechanism to transfer state
> > > from a follower to a prospective leader.
> > >
> > > -Flavio
> > >
> > > > -----Original Message-----
> > > > From: Benjamin Reed [mailto:breed@apache.org]
> > > > Sent: 25 November 2013 01:21
> > > > To: user@zookeeper.apache.org
> > > > Subject: Re: Disqualify a node from leader election
> > > >
> > > > camille really has the right solution. you have to let it become the
> > > leader and
> > > > then kill it. here is why:
> > > >
> > > > lets says you have servers: A, B, C, D, and E and A is the node that
> > > > you
> > > don't
> > > > want to be the leader. let's also say that C is a leader and commits
> > > transaction
> > > > x on A, B, and C but before D and E get x B and C fail. now A is the
> > > > only surviving node with x, so unless A can become the leader, at
> > > > least temporarily, D and E will never get x. if you follow Camille's
> > > suggestion,
> > > you
> > > > let A become the leader (since it has the most recent
> > > > transaction) and D and E will sync with A and get x. now if you
> > > > restart
> > > A,
> > > > leader election will run again and D or E can be elected leader.
> > > > this
> > > does
> > > also
> > > > show that you want the less desirable nodes to have lower server ids
> > > > so
> > > that
> > > > there will be less chance of them becoming leader if there is a "tie"
> > > (other
> > > > nodes are just as uptodate as them).
> > > >
> > > > ben
> > > >
> > > >
> > > > On Sun, Nov 24, 2013 at 5:09 PM, Owen Kim <oh...@gmail.com>
> > wrote:
> > > >
> > > > > Observers can't vote in leader election, though right? I'm not
> > > > > sure the loss of fault tolerance would be worth it.
> > > > >
> > > > > The scenario is that I have a 5-node cluster but one node is in an
> > > > > network partition that gets DOSed around it. When this happens and
> > > > > it's leader, I see huge performance degradation. The real solution
> > > > > is obviously to move the node off that network but I wondered if
> > > > > there was an easy way to keep it from being leader in its
> configuration.
> > > > >
> > >
> > >
>
>

RE: Disqualify a node from leader election

[FPJ <fp...@yahoo.com>]

The problem with this solution is that setting the weight of E to zero will
make its acknowledgments not count for committing txns or anything in the
system that requires a quorum. Also, I don't think it prevents E from being
elected, E is still eligible because it is a participant (but with vote
zero).

-Flavio

> -----Original Message-----
> From: Kuba Lekstan [mailto:kuebzky@gmail.com]
> Sent: 25 November 2013 15:55
> To: user@zookeeper.apache.org
> Subject: Re: Disqualify a node from leader election
> 
> If I understand you correctly as you want the specific node never become
> the leader.
> 
> The best solution would be to configure your cluster into 2 groups, put
first 4
> servers into group 1 with weight 10 and 5th server into group 2 with
weight 0.
> This way 5th servers never gets elected as a leader.
> 
> http://zookeeper.apache.org/doc/trunk/zookeeperHierarchicalQuorums.ht
> ml
> 
> 
> 2013/11/25 FPJ <fp...@yahoo.com>
> 
> > D and E can only become a leader if they can form a quorum with B
> > and/or C (assuming you've taken A down). In the case A, D, E are the
> > only 3 servers able to talk to each other, killing A is going to make
> > you lose quorum, and if you bring A back up, it might be elected
> > again, so you might end up in this cycle until the net partition is
healed or B
> and/or C are up.
> >
> > I suppose we could have an option that tells servers to ignore a given
> > server when electing a leader, but it is not entirely trivial because
> > A in the example Ben gave might be the only available server that has
> > the latest committed txn. We would need a mechanism to transfer state
> > from a follower to a prospective leader.
> >
> > -Flavio
> >
> > > -----Original Message-----
> > > From: Benjamin Reed [mailto:breed@apache.org]
> > > Sent: 25 November 2013 01:21
> > > To: user@zookeeper.apache.org
> > > Subject: Re: Disqualify a node from leader election
> > >
> > > camille really has the right solution. you have to let it become the
> > leader and
> > > then kill it. here is why:
> > >
> > > lets says you have servers: A, B, C, D, and E and A is the node that
> > > you
> > don't
> > > want to be the leader. let's also say that C is a leader and commits
> > transaction
> > > x on A, B, and C but before D and E get x B and C fail. now A is the
> > > only surviving node with x, so unless A can become the leader, at
> > > least temporarily, D and E will never get x. if you follow Camille's
> > suggestion,
> > you
> > > let A become the leader (since it has the most recent
> > > transaction) and D and E will sync with A and get x. now if you
> > > restart
> > A,
> > > leader election will run again and D or E can be elected leader.
> > > this
> > does
> > also
> > > show that you want the less desirable nodes to have lower server ids
> > > so
> > that
> > > there will be less chance of them becoming leader if there is a "tie"
> > (other
> > > nodes are just as uptodate as them).
> > >
> > > ben
> > >
> > >
> > > On Sun, Nov 24, 2013 at 5:09 PM, Owen Kim <oh...@gmail.com>
> wrote:
> > >
> > > > Observers can't vote in leader election, though right? I'm not
> > > > sure the loss of fault tolerance would be worth it.
> > > >
> > > > The scenario is that I have a 5-node cluster but one node is in an
> > > > network partition that gets DOSed around it. When this happens and
> > > > it's leader, I see huge performance degradation. The real solution
> > > > is obviously to move the node off that network but I wondered if
> > > > there was an easy way to keep it from being leader in its
configuration.
> > > >
> >
> >

Re: Disqualify a node from leader election

[Kuba Lekstan <ku...@gmail.com>]

If I understand you correctly as you want the specific node never become
the leader.

The best solution would be to configure your cluster into 2 groups, put
first 4 servers into group 1 with weight 10 and 5th server into group 2
with weight 0.
This way 5th servers never gets elected as a leader.

http://zookeeper.apache.org/doc/trunk/zookeeperHierarchicalQuorums.html


2013/11/25 FPJ <fp...@yahoo.com>

> D and E can only become a leader if they can form a quorum with B and/or C
> (assuming you've taken A down). In the case A, D, E are the only 3 servers
> able to talk to each other, killing A is going to make you lose quorum, and
> if you bring A back up, it might be elected again, so you might end up in
> this cycle until the net partition is healed or B and/or C are up.
>
> I suppose we could have an option that tells servers to ignore a given
> server when electing a leader, but it is not entirely trivial because A in
> the example Ben gave might be the only available server that has the latest
> committed txn. We would need a mechanism to transfer state from a follower
> to a prospective leader.
>
> -Flavio
>
> > -----Original Message-----
> > From: Benjamin Reed [mailto:breed@apache.org]
> > Sent: 25 November 2013 01:21
> > To: user@zookeeper.apache.org
> > Subject: Re: Disqualify a node from leader election
> >
> > camille really has the right solution. you have to let it become the
> leader and
> > then kill it. here is why:
> >
> > lets says you have servers: A, B, C, D, and E and A is the node that you
> don't
> > want to be the leader. let's also say that C is a leader and commits
> transaction
> > x on A, B, and C but before D and E get x B and C fail. now A is the only
> > surviving node with x, so unless A can become the leader, at least
> > temporarily, D and E will never get x. if you follow Camille's
> suggestion,
> you
> > let A become the leader (since it has the most recent
> > transaction) and D and E will sync with A and get x. now if you restart
> A,
> > leader election will run again and D or E can be elected leader. this
> does
> also
> > show that you want the less desirable nodes to have lower server ids so
> that
> > there will be less chance of them becoming leader if there is a "tie"
> (other
> > nodes are just as uptodate as them).
> >
> > ben
> >
> >
> > On Sun, Nov 24, 2013 at 5:09 PM, Owen Kim <oh...@gmail.com> wrote:
> >
> > > Observers can't vote in leader election, though right? I'm not sure
> > > the loss of fault tolerance would be worth it.
> > >
> > > The scenario is that I have a 5-node cluster but one node is in an
> > > network partition that gets DOSed around it. When this happens and
> > > it's leader, I see huge performance degradation. The real solution is
> > > obviously to move the node off that network but I wondered if there
> > > was an easy way to keep it from being leader in its configuration.
> > >
>
>

RE: Disqualify a node from leader election

[FPJ <fp...@yahoo.com>]

D and E can only become a leader if they can form a quorum with B and/or C
(assuming you've taken A down). In the case A, D, E are the only 3 servers
able to talk to each other, killing A is going to make you lose quorum, and
if you bring A back up, it might be elected again, so you might end up in
this cycle until the net partition is healed or B and/or C are up.

I suppose we could have an option that tells servers to ignore a given
server when electing a leader, but it is not entirely trivial because A in
the example Ben gave might be the only available server that has the latest
committed txn. We would need a mechanism to transfer state from a follower
to a prospective leader.

-Flavio

> -----Original Message-----
> From: Benjamin Reed [mailto:breed@apache.org]
> Sent: 25 November 2013 01:21
> To: user@zookeeper.apache.org
> Subject: Re: Disqualify a node from leader election
> 
> camille really has the right solution. you have to let it become the
leader and
> then kill it. here is why:
> 
> lets says you have servers: A, B, C, D, and E and A is the node that you
don't
> want to be the leader. let's also say that C is a leader and commits
transaction
> x on A, B, and C but before D and E get x B and C fail. now A is the only
> surviving node with x, so unless A can become the leader, at least
> temporarily, D and E will never get x. if you follow Camille's suggestion,
you
> let A become the leader (since it has the most recent
> transaction) and D and E will sync with A and get x. now if you restart A,
> leader election will run again and D or E can be elected leader. this does
also
> show that you want the less desirable nodes to have lower server ids so
that
> there will be less chance of them becoming leader if there is a "tie"
(other
> nodes are just as uptodate as them).
> 
> ben
> 
> 
> On Sun, Nov 24, 2013 at 5:09 PM, Owen Kim <oh...@gmail.com> wrote:
> 
> > Observers can't vote in leader election, though right? I'm not sure
> > the loss of fault tolerance would be worth it.
> >
> > The scenario is that I have a 5-node cluster but one node is in an
> > network partition that gets DOSed around it. When this happens and
> > it's leader, I see huge performance degradation. The real solution is
> > obviously to move the node off that network but I wondered if there
> > was an easy way to keep it from being leader in its configuration.
> >

Re: Disqualify a node from leader election

[Benjamin Reed <br...@apache.org>]

camille really has the right solution. you have to let it become the leader
and then kill it. here is why:

lets says you have servers: A, B, C, D, and E and A is the node that you
don't want to be the leader. let's also say that C is a leader and commits
transaction x on A, B, and C but before D and E get x B and C fail. now A
is the only surviving node with x, so unless A can become the leader, at
least temporarily, D and E will never get x. if you follow Camille's
suggestion, you let A become the leader (since it has the most recent
transaction) and D and E will sync with A and get x. now if you restart A,
leader election will run again and D or E can be elected leader. this does
also show that you want the less desirable nodes to have lower server ids
so that there will be less chance of them becoming leader if there is a
"tie" (other nodes are just as uptodate as them).

ben


On Sun, Nov 24, 2013 at 5:09 PM, Owen Kim <oh...@gmail.com> wrote:

> Observers can't vote in leader election, though right? I'm not sure the
> loss of fault tolerance would be worth it.
>
> The scenario is that I have a 5-node cluster but one node is in an network
> partition that gets DOSed around it. When this happens and it's leader, I
> see huge performance degradation. The real solution is obviously to move
> the node off that network but I wondered if there was an easy way to keep
> it from being leader in its configuration.
>

Re: Disqualify a node from leader election

[Owen Kim <oh...@gmail.com>]

Observers can't vote in leader election, though right? I'm not sure the
loss of fault tolerance would be worth it.

The scenario is that I have a 5-node cluster but one node is in an network
partition that gets DOSed around it. When this happens and it's leader, I
see huge performance degradation. The real solution is obviously to move
the node off that network but I wondered if there was an easy way to keep
it from being leader in its configuration.

RE: Disqualify a node from leader election

[FPJ <fp...@yahoo.com>]

Would it work for you to make 2 nodes observers?

-Flavio

> -----Original Message-----
> From: Owen Kim [mailto:ohechkay@gmail.com]
> Sent: 22 November 2013 02:11
> To: user
> Subject: Disqualify a node from leader election
> 
> I have a 5-node cluster and have a node that I want to participate in
leader
> election for quorum but never be a leader candidate. Is it possible to
> configure the cluster to never nominate it?

Re: Disqualify a node from leader election

[Camille Fournier <ca...@apache.org>]

No. I recommend monitoring the node and should it become leader, kill the
node and let it reelect.


On Thu, Nov 21, 2013 at 9:10 PM, Owen Kim <oh...@gmail.com> wrote:

> I have a 5-node cluster and have a node that I want to participate in
> leader election for quorum but never be a leader candidate. Is it possible
> to configure the cluster to never nominate it?
>