You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by Todd Nine <to...@spidertracks.co.nz> on 2010/08/25 23:54:14 UTC
What roles do "even" nodes play in the ensamble
Hey guys,
Forgive me if this is documented somewhere, but I can't find an
answer. Our application is not enormous, so we will be using 4
application nodes that will also initially run Zookeeper. As our load
increases, Zookeeper will be moved to nodes that only run ZK and no
other processes. Given that we will initially only have 4 nodes in our
cluster and I have a few questions around the semantics of an even
number of nodes.
1. When the 4th ZK node joins the cluster, does it take on the observer
role since a quorum cannot be reached with the new node? Can I still
connect my clients to it and create/remove nodes and receive events?
2. In the event 1 of the 3 voting nodes fails, will this 4th node become
a voting member of the ensemble?
3. When a new node comes online, it may have a different ip than the
previous node. Do I need to update all node configurations and perform
a rolling restart, or will simply connecting the new node to the
existing ensemble make all nodes aware it is running?
Thanks,
Todd
Re: What roles do "even" nodes play in the ensamble
Posted by Todd Nine <to...@spidertracks.co.nz>.
Awesome, thanks guys. Your patience and input is greatly appreciated.
On Wed, 2010-08-25 at 21:30 -0700, Henry Robinson wrote:
> Todd -
>
>
>
> No, this is not the case. There are no 'backup' or 'failover' nodes in
> ZooKeeper. All servers that can vote are working as part of the
> cluster until they fail. You need a majority of your voting servers
> alive.
>
>
> If you have three servers, a majority is of size two. The number of
> nodes that can fail before a majority is no longer alive is one.
> If you have four servers, a majority is of size three. The number of
> nodes that can fail before a majority is no longer alive is one.
> If you have five servers, a majority is of size three. The number of
> nodes that can fail before a majority is no longer alive is two.
>
>
> This is why four servers is worse than three for availability. In both
> cases, two servers have to fail before the cluster is no longer
> available. However if failures are independently distributed, this is
> more likely to happen in a cluster of four nodes than a cluster of
> three (think of it as 'more things available to go wrong').
>
>
> If you have four servers and one dies, the 'majority' that still needs
> to be alive is still three - it doesn't drop down to two. The majority
> is of all voting servers, alive or dead.
>
>
> Hope this helps -
>
>
> Henry
>
>
> On 25 August 2010 21:01, Todd Nine <to...@spidertracks.co.nz> wrote:
>
> Thanks Dave. I've been using Cassandra, so I'm trying to get
> my head
> around the configuration/operational differences with ZK. You
> state
> that using 4 would actually decrease my reliability. Can you
> explain
> that further? I was under the impression that a 4th node
> would act as a
> non voting read only node until one of the other 3 fails. I
> thought
> that this extra node would give me some breathing room by
> allowing any
> node to fail and still have 3 voting nodes. Is this not the
> case?
>
> Thanks,
>
> Todd
>
>
>
>
>
>
> On Wed, 2010-08-25 at 21:13 -0600, Ted Dunning wrote:
>
> > Just use 3 nodes. Life will be better.
> >
> >
> >
> > You can configure the fourth node in the event of one of the
> first
> > three failing and bring it on line. Then you can
> re-configure and
> > restart each of the others one at a time. This gives you
> flexibility
> > because you have 4 nodes, but doesn't decrease your
> reliability the
> > way that using a four node cluster would. If you need to do
> > maintenance on one node, just configure that node out as if
> it had
> > failed.
> >
> >
> > On Wed, Aug 25, 2010 at 4:26 PM, Dave Wright
> <wr...@gmail.com>
> > wrote:
> >
> > You can certainly serve more reads with a 4th node,
> but I'm
> > not sure
> > what you mean by "it won't have a voting role". It
> still
> > participates
> > in voting for leaders as do all non-observers
> regardless of
> > whether it
> > is an even or odd number. With zookeeper there is no
> voting on
> > each
> > transaction, only leader changes.
> >
> > -Dave Wright
> >
> >
> >
> > On Wed, Aug 25, 2010 at 6:22 PM, Todd Nine
> > <to...@spidertracks.co.nz> wrote:
> > > Do I get any read performance increase (similar to
> an
> > observer) since
> > > the node will not have a voting role?
> > >
> > >
> >
> >
> >
> >
>
>
>
>
>
> --
> Henry Robinson
> Software Engineer
> Cloudera
> 415-994-6679
>
Re: What roles do "even" nodes play in the ensamble
Posted by Henry Robinson <he...@cloudera.com>.
Todd -
No, this is not the case. There are no 'backup' or 'failover' nodes in
ZooKeeper. All servers that can vote are working as part of the cluster
until they fail. You need a majority of your voting servers alive.
If you have three servers, a majority is of size two. The number of nodes
that can fail before a majority is no longer alive is one.
If you have four servers, a majority is of size three. The number of nodes
that can fail before a majority is no longer alive is one.
If you have five servers, a majority is of size three. The number of nodes
that can fail before a majority is no longer alive is two.
This is why four servers is worse than three for availability. In both
cases, two servers have to fail before the cluster is no longer available.
However if failures are independently distributed, this is more likely to
happen in a cluster of four nodes than a cluster of three (think of it as
'more things available to go wrong').
If you have four servers and one dies, the 'majority' that still needs to be
alive is still three - it doesn't drop down to two. The majority is of all
voting servers, alive or dead.
Hope this helps -
Henry
On 25 August 2010 21:01, Todd Nine <to...@spidertracks.co.nz> wrote:
> Thanks Dave. I've been using Cassandra, so I'm trying to get my head
> around the configuration/operational differences with ZK. You state
> that using 4 would actually decrease my reliability. Can you explain
> that further? I was under the impression that a 4th node would act as a
> non voting read only node until one of the other 3 fails. I thought
> that this extra node would give me some breathing room by allowing any
> node to fail and still have 3 voting nodes. Is this not the case?
>
> Thanks,
>
> Todd
>
>
>
>
> On Wed, 2010-08-25 at 21:13 -0600, Ted Dunning wrote:
>
> > Just use 3 nodes. Life will be better.
> >
> >
> >
> > You can configure the fourth node in the event of one of the first
> > three failing and bring it on line. Then you can re-configure and
> > restart each of the others one at a time. This gives you flexibility
> > because you have 4 nodes, but doesn't decrease your reliability the
> > way that using a four node cluster would. If you need to do
> > maintenance on one node, just configure that node out as if it had
> > failed.
> >
> >
> > On Wed, Aug 25, 2010 at 4:26 PM, Dave Wright <wr...@gmail.com>
> > wrote:
> >
> > You can certainly serve more reads with a 4th node, but I'm
> > not sure
> > what you mean by "it won't have a voting role". It still
> > participates
> > in voting for leaders as do all non-observers regardless of
> > whether it
> > is an even or odd number. With zookeeper there is no voting on
> > each
> > transaction, only leader changes.
> >
> > -Dave Wright
> >
> >
> >
> > On Wed, Aug 25, 2010 at 6:22 PM, Todd Nine
> > <to...@spidertracks.co.nz> wrote:
> > > Do I get any read performance increase (similar to an
> > observer) since
> > > the node will not have a voting role?
> > >
> > >
> >
> >
> >
> >
>
--
Henry Robinson
Software Engineer
Cloudera
415-994-6679
Re: What roles do "even" nodes play in the ensamble
Posted by Todd Nine <to...@spidertracks.co.nz>.
Thanks Dave. I've been using Cassandra, so I'm trying to get my head
around the configuration/operational differences with ZK. You state
that using 4 would actually decrease my reliability. Can you explain
that further? I was under the impression that a 4th node would act as a
non voting read only node until one of the other 3 fails. I thought
that this extra node would give me some breathing room by allowing any
node to fail and still have 3 voting nodes. Is this not the case?
Thanks,
Todd
On Wed, 2010-08-25 at 21:13 -0600, Ted Dunning wrote:
> Just use 3 nodes. Life will be better.
>
>
>
> You can configure the fourth node in the event of one of the first
> three failing and bring it on line. Then you can re-configure and
> restart each of the others one at a time. This gives you flexibility
> because you have 4 nodes, but doesn't decrease your reliability the
> way that using a four node cluster would. If you need to do
> maintenance on one node, just configure that node out as if it had
> failed.
>
>
> On Wed, Aug 25, 2010 at 4:26 PM, Dave Wright <wr...@gmail.com>
> wrote:
>
> You can certainly serve more reads with a 4th node, but I'm
> not sure
> what you mean by "it won't have a voting role". It still
> participates
> in voting for leaders as do all non-observers regardless of
> whether it
> is an even or odd number. With zookeeper there is no voting on
> each
> transaction, only leader changes.
>
> -Dave Wright
>
>
>
> On Wed, Aug 25, 2010 at 6:22 PM, Todd Nine
> <to...@spidertracks.co.nz> wrote:
> > Do I get any read performance increase (similar to an
> observer) since
> > the node will not have a voting role?
> >
> >
>
>
>
>
Re: What roles do "even" nodes play in the ensamble
Posted by Ted Dunning <te...@gmail.com>.
Just use 3 nodes. Life will be better.
You can configure the fourth node in the event of one of the first three
failing and bring it on line. Then you can re-configure and restart each of
the others one at a time. This gives you flexibility because you have 4
nodes, but doesn't decrease your reliability the way that using a four node
cluster would. If you need to do maintenance on one node, just configure
that node out as if it had failed.
On Wed, Aug 25, 2010 at 4:26 PM, Dave Wright <wr...@gmail.com> wrote:
> You can certainly serve more reads with a 4th node, but I'm not sure
> what you mean by "it won't have a voting role". It still participates
> in voting for leaders as do all non-observers regardless of whether it
> is an even or odd number. With zookeeper there is no voting on each
> transaction, only leader changes.
>
> -Dave Wright
>
> On Wed, Aug 25, 2010 at 6:22 PM, Todd Nine <to...@spidertracks.co.nz>
> wrote:
> > Do I get any read performance increase (similar to an observer) since
> > the node will not have a voting role?
> >
> >
>
Re: What roles do "even" nodes play in the ensamble
Posted by Dave Wright <wr...@gmail.com>.
You can certainly serve more reads with a 4th node, but I'm not sure
what you mean by "it won't have a voting role". It still participates
in voting for leaders as do all non-observers regardless of whether it
is an even or odd number. With zookeeper there is no voting on each
transaction, only leader changes.
-Dave Wright
On Wed, Aug 25, 2010 at 6:22 PM, Todd Nine <to...@spidertracks.co.nz> wrote:
> Do I get any read performance increase (similar to an observer) since
> the node will not have a voting role?
>
>
Re: What roles do "even" nodes play in the ensamble
Posted by Todd Nine <to...@spidertracks.co.nz>.
Do I get any read performance increase (similar to an observer) since
the node will not have a voting role?
On Wed, 2010-08-25 at 15:18 -0700, Henry Robinson wrote:
> Dave is correct - if you have N nodes you need (N/2) + 1 votes (i.e. a
> majority) in the standard case to get a vote to pass.
>
> Adding a fourth voting node to a three node cluster will cause the size of a
> majority to jump from 2 to 3. The number of nodes that need to fail before
> you can no longer get a majority is 2 in both cases - so you don't get any
> reliability for adding a new voting node to a odd-numbered cluster.
>
> The new node will always act as a voter unless you explicitly configure it
> as an observer.
>
> Henry
>
> On 25 August 2010 15:11, Dave Wright <wr...@gmail.com> wrote:
>
> > I'm not an expert on voting, so there may be a better answer, but from my
> > understanding all 4 nodes participate in the voting and you need a majority
> > of 3 to elect a leader.
> >
> > -Dave
> >
> > On Wed, Aug 25, 2010 at 6:09 PM, Todd Nine <to...@spidertracks.co.nz>
> > wrote:
> >
> > > Thanks for that Dave. If I do not configure it as an observer just a
> > > normal member, what will the last even node to join do?
> > >
> > >
> > > 1. Will it participate as a voter on startup? (I'm assuming not, just
> > read
> > > only)
> > >
> > > 2. If one of the voter nodes 1 through 3 dies, does it become a voter?
> > >
> > >
> > > todd
> > > SENIOR SOFTWARE ENGINEER
> > >
> > > todd nine| spidertracks ltd | 117a the square
> > > po box 5203 | palmerston north 4441 | new zealand
> > > P: +64 6 353 3395 | M: +64 210 255 8576
> > > E: todd@spidertracks.co.nz W: www.spidertracks.com
> > >
> > >
> > >
> > >
> > >
> > > On Wed, 2010-08-25 at 17:57 -0400, Dave Wright wrote:
> > >
> > > >
> > > > 1. When the 4th ZK node joins the cluster, does it take on the observer
> > > > role since a quorum cannot be reached with the new node? Can I still
> > > > connect my clients to it and create/remove nodes and receive events?
> > >
> > > No, it joins as a normal member unless you've configured it as an
> > > observer. Note that with 4 nodes you now need 3 running to get a
> > > majority, which is why even numbers aren't recommended.
> > >
> > > >
> > > >
> > > > 2. In the event 1 of the 3 voting nodes fails, will this 4th node
> > become
> > > > a voting member of the ensemble?
> > >
> > > If configured as an observer it remains an observer.
> > >
> > > >
> > > > 3. When a new node comes online, it may have a different ip than the
> > > > previous node. Do I need to update all node configurations and perform
> > > > a rolling restart, or will simply connecting the new node to the
> > > > existing ensemble make all nodes aware it is running?
> > >
> > > Unfortunately ZK doesn't have any kind of dynamic configuration like
> > > that currently. You need to update all the config files and restart
> > > the ensemble.
> > >
> > > -Dave Wright
> > >
> > >
> >
>
>
>
Re: What roles do "even" nodes play in the ensamble
Posted by Henry Robinson <he...@cloudera.com>.
Dave is correct - if you have N nodes you need (N/2) + 1 votes (i.e. a
majority) in the standard case to get a vote to pass.
Adding a fourth voting node to a three node cluster will cause the size of a
majority to jump from 2 to 3. The number of nodes that need to fail before
you can no longer get a majority is 2 in both cases - so you don't get any
reliability for adding a new voting node to a odd-numbered cluster.
The new node will always act as a voter unless you explicitly configure it
as an observer.
Henry
On 25 August 2010 15:11, Dave Wright <wr...@gmail.com> wrote:
> I'm not an expert on voting, so there may be a better answer, but from my
> understanding all 4 nodes participate in the voting and you need a majority
> of 3 to elect a leader.
>
> -Dave
>
> On Wed, Aug 25, 2010 at 6:09 PM, Todd Nine <to...@spidertracks.co.nz>
> wrote:
>
> > Thanks for that Dave. If I do not configure it as an observer just a
> > normal member, what will the last even node to join do?
> >
> >
> > 1. Will it participate as a voter on startup? (I'm assuming not, just
> read
> > only)
> >
> > 2. If one of the voter nodes 1 through 3 dies, does it become a voter?
> >
> >
> > todd
> > SENIOR SOFTWARE ENGINEER
> >
> > todd nine| spidertracks ltd | 117a the square
> > po box 5203 | palmerston north 4441 | new zealand
> > P: +64 6 353 3395 | M: +64 210 255 8576
> > E: todd@spidertracks.co.nz W: www.spidertracks.com
> >
> >
> >
> >
> >
> > On Wed, 2010-08-25 at 17:57 -0400, Dave Wright wrote:
> >
> > >
> > > 1. When the 4th ZK node joins the cluster, does it take on the observer
> > > role since a quorum cannot be reached with the new node? Can I still
> > > connect my clients to it and create/remove nodes and receive events?
> >
> > No, it joins as a normal member unless you've configured it as an
> > observer. Note that with 4 nodes you now need 3 running to get a
> > majority, which is why even numbers aren't recommended.
> >
> > >
> > >
> > > 2. In the event 1 of the 3 voting nodes fails, will this 4th node
> become
> > > a voting member of the ensemble?
> >
> > If configured as an observer it remains an observer.
> >
> > >
> > > 3. When a new node comes online, it may have a different ip than the
> > > previous node. Do I need to update all node configurations and perform
> > > a rolling restart, or will simply connecting the new node to the
> > > existing ensemble make all nodes aware it is running?
> >
> > Unfortunately ZK doesn't have any kind of dynamic configuration like
> > that currently. You need to update all the config files and restart
> > the ensemble.
> >
> > -Dave Wright
> >
> >
>
--
Henry Robinson
Software Engineer
Cloudera
415-994-6679
Re: What roles do "even" nodes play in the ensamble
Posted by Dave Wright <wr...@gmail.com>.
I'm not an expert on voting, so there may be a better answer, but from my
understanding all 4 nodes participate in the voting and you need a majority
of 3 to elect a leader.
-Dave
On Wed, Aug 25, 2010 at 6:09 PM, Todd Nine <to...@spidertracks.co.nz> wrote:
> Thanks for that Dave. If I do not configure it as an observer just a
> normal member, what will the last even node to join do?
>
>
> 1. Will it participate as a voter on startup? (I'm assuming not, just read
> only)
>
> 2. If one of the voter nodes 1 through 3 dies, does it become a voter?
>
>
> todd
> SENIOR SOFTWARE ENGINEER
>
> todd nine| spidertracks ltd | 117a the square
> po box 5203 | palmerston north 4441 | new zealand
> P: +64 6 353 3395 | M: +64 210 255 8576
> E: todd@spidertracks.co.nz W: www.spidertracks.com
>
>
>
>
>
> On Wed, 2010-08-25 at 17:57 -0400, Dave Wright wrote:
>
> >
> > 1. When the 4th ZK node joins the cluster, does it take on the observer
> > role since a quorum cannot be reached with the new node? Can I still
> > connect my clients to it and create/remove nodes and receive events?
>
> No, it joins as a normal member unless you've configured it as an
> observer. Note that with 4 nodes you now need 3 running to get a
> majority, which is why even numbers aren't recommended.
>
> >
> >
> > 2. In the event 1 of the 3 voting nodes fails, will this 4th node become
> > a voting member of the ensemble?
>
> If configured as an observer it remains an observer.
>
> >
> > 3. When a new node comes online, it may have a different ip than the
> > previous node. Do I need to update all node configurations and perform
> > a rolling restart, or will simply connecting the new node to the
> > existing ensemble make all nodes aware it is running?
>
> Unfortunately ZK doesn't have any kind of dynamic configuration like
> that currently. You need to update all the config files and restart
> the ensemble.
>
> -Dave Wright
>
>
Re: What roles do "even" nodes play in the ensamble
Posted by Todd Nine <to...@spidertracks.co.nz>.
Thanks for that Dave. If I do not configure it as an observer just a
normal member, what will the last even node to join do?
1. Will it participate as a voter on startup? (I'm assuming not, just
read only)
2. If one of the voter nodes 1 through 3 dies, does it become a voter?
todd
SENIOR SOFTWARE ENGINEER
todd nine| spidertracks ltd | 117a the square
po box 5203 | palmerston north 4441 | new zealand
P: +64 6 353 3395 | M: +64 210 255 8576
E: todd@spidertracks.co.nz W: www.spidertracks.com
On Wed, 2010-08-25 at 17:57 -0400, Dave Wright wrote:
> >
> > 1. When the 4th ZK node joins the cluster, does it take on the observer
> > role since a quorum cannot be reached with the new node? Can I still
> > connect my clients to it and create/remove nodes and receive events?
>
> No, it joins as a normal member unless you've configured it as an
> observer. Note that with 4 nodes you now need 3 running to get a
> majority, which is why even numbers aren't recommended.
>
> >
> >
> > 2. In the event 1 of the 3 voting nodes fails, will this 4th node become
> > a voting member of the ensemble?
>
> If configured as an observer it remains an observer.
>
> >
> > 3. When a new node comes online, it may have a different ip than the
> > previous node. Do I need to update all node configurations and perform
> > a rolling restart, or will simply connecting the new node to the
> > existing ensemble make all nodes aware it is running?
>
> Unfortunately ZK doesn't have any kind of dynamic configuration like
> that currently. You need to update all the config files and restart
> the ensemble.
>
> -Dave Wright
Re: What roles do "even" nodes play in the ensamble
Posted by Dave Wright <wr...@gmail.com>.
>
> 1. When the 4th ZK node joins the cluster, does it take on the observer
> role since a quorum cannot be reached with the new node? Can I still
> connect my clients to it and create/remove nodes and receive events?
No, it joins as a normal member unless you've configured it as an
observer. Note that with 4 nodes you now need 3 running to get a
majority, which is why even numbers aren't recommended.
>
>
> 2. In the event 1 of the 3 voting nodes fails, will this 4th node become
> a voting member of the ensemble?
If configured as an observer it remains an observer.
>
> 3. When a new node comes online, it may have a different ip than the
> previous node. Do I need to update all node configurations and perform
> a rolling restart, or will simply connecting the new node to the
> existing ensemble make all nodes aware it is running?
Unfortunately ZK doesn't have any kind of dynamic configuration like
that currently. You need to update all the config files and restart
the ensemble.
-Dave Wright