You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by Denis Samoilov <sa...@gmail.com> on 2014/10/21 21:03:50 UTC
cross DC setup - is it Ok for ZK?
hi,
Could you please help to understand the following setup: we have two
datacenters and want to setup ZK cluster so it will use servers (ZK servers
not clients) in both: like 3 ZK servers in DC1 and 4 ZK servers in DC2. We
sometime do maintenance in one or other DC. So ZK will completely lose
replicas in one of the DC for several hours. E.g. if DC2 is under
maintenance ZK will have only 3 out of 7 nodes and these 3 nodes supposed
to receive writes.
The questions:
1) is it Ok for ZK to have such setup?
2) will ZK catch up after losing 4 Servers and getting them back in some
time? (this will be a majority actually :) )
3) what is right number of nodes, is 5 sufficient : 2 + 3?
Latency between DCs is pretty low (DCs are close to each other).
Thank you for any advice.
-Denis
Re: cross DC setup - is it Ok for ZK?
Posted by Flavio Junqueira <fp...@yahoo.com.INVALID>.
Not for two data centers, you need an intersection somewhere. With these simple hierarchies you can reduce the number of members of a quorum. In the case I described before, you can do it with 4 out of 9 instead of 5 out of 9. It isn't any 4 out 9, though.
In the case you're interested, there is some more detail here: http://sysnet.ucsd.edu/~flavio/Docs/hotdep05.pdf
-Flavio
On Tuesday, October 21, 2014 11:51 PM, Camille Fournier <ca...@apache.org> wrote:
I must admit that I tried to reason through this when I originally did all this work and didn't see how you could make it so that you could split the servers across 2 data centers and guarantee that you will always be up if one whole data center is down. Are you saying that you can do that with these hierarchies?
On Tue, Oct 21, 2014 at 6:35 PM, Flavio Junqueira <fp...@yahoo.com.invalid> wrote:
Hierarchical quorums don't rely on strict majorities. Quorums are formed by taking majorities from a majority of groups, so if you have 3 groups of 3 servers, you have a quorum with 4 servers, 2 from each of 2 distinct groups. This is a different way of doing quorums in ZooKeeper if grouping makes sense in your scenario, like when you have multiple colos.
-Flavio
On Tuesday, October 21, 2014 10:15 PM, Camille Fournier <ca...@apache.org> wrote:
>
>
>You'll have to ask Flavio because I don't really understand what he's
>saying there, tbh. You have to have (n+1)/2 nodes available and
>communicating to each other to maintain a live quorum (so, in your case,
>you must have 4 nodes available at all times to maintain a live quorum).
>
>
>On Tue, Oct 21, 2014 at 5:08 PM, Denis Samoilov <sa...@gmail.com> wrote:
>
>> Camille, thank you very much! Very interesting read. I also found a thread
>> where you participated three years ago and found one particular comment a
>> bit confusing *"One quick comment. We do not require majority quorums in
>> ZooKeeper, and one reason we implemented this feature was exactly to
>> enable more flexibility in deployments with multiple data centers*" (
>>
>> http://mail-archives.apache.org/mod_mbox/zookeeper-user/201109.mbox/%3C0B4CC52A-939E-4896-A269-50DC31E20AA6@yahoo-inc.com%3E
>> )
>> but this potentially contradicts FAQ: *"if the leader is in the non-quorum
>> side of the partition, that side of the partition will recognize that it no
>> longer has a quorum of the ensemble"* (
>> https://cwiki.apache.org/confluence/display/ZOOKEEPER/FailureScenarios).
>>
>> Where is the truth? :)
>>
>> On Tue, Oct 21, 2014 at 12:35 PM, Camille Fournier <ca...@apache.org>
>> wrote:
>>
>> > I have a blog post on this topic:
>> >
>> >
>> http://whilefalse.blogspot.com/2012/12/building-global-highly-available.html
>> >
>> > I think you will find it helpful.
>> > The short answer is: the scheme you have proposed will cause the ZK to be
>> > unavailable when you do maintenance on the data center with 4 quorum
>> > members.
>> >
>> > Best,
>> > C
>> >
>> > On Tue, Oct 21, 2014 at 3:03 PM, Denis Samoilov <sa...@gmail.com>
>> > wrote:
>> >
>> > > hi,
>> > >
>> > > Could you please help to understand the following setup: we have two
>> > > datacenters and want to setup ZK cluster so it will use servers (ZK
>> > servers
>> > > not clients) in both: like 3 ZK servers in DC1 and 4 ZK servers in DC2.
>> > We
>> > > sometime do maintenance in one or other DC. So ZK will completely lose
>> > > replicas in one of the DC for several hours. E.g. if DC2 is under
>> > > maintenance ZK will have only 3 out of 7 nodes and these 3 nodes
>> supposed
>> > > to receive writes.
>> > >
>> > > The questions:
>> > > 1) is it Ok for ZK to have such setup?
>> > > 2) will ZK catch up after losing 4 Servers and getting them back in
>> some
>> > > time? (this will be a majority actually :) )
>> > > 3) what is right number of nodes, is 5 sufficient : 2 + 3?
>> > >
>> > > Latency between DCs is pretty low (DCs are close to each other).
>> > >
>> > >
>> > > Thank you for any advice.
>> > >
>> > > -Denis
>> > >
>> >
>>
>
>
>
Re: cross DC setup - is it Ok for ZK?
Posted by Camille Fournier <ca...@apache.org>.
I must admit that I tried to reason through this when I originally did all
this work and didn't see how you could make it so that you could split the
servers across 2 data centers and guarantee that you will always be up if
one whole data center is down. Are you saying that you can do that with
these hierarchies?
On Tue, Oct 21, 2014 at 6:35 PM, Flavio Junqueira <
fpjunqueira@yahoo.com.invalid> wrote:
> Hierarchical quorums don't rely on strict majorities. Quorums are formed
> by taking majorities from a majority of groups, so if you have 3 groups of
> 3 servers, you have a quorum with 4 servers, 2 from each of 2 distinct
> groups. This is a different way of doing quorums in ZooKeeper if grouping
> makes sense in your scenario, like when you have multiple colos.
>
> -Flavio
>
>
>
> On Tuesday, October 21, 2014 10:15 PM, Camille Fournier <
> camille@apache.org> wrote:
>
>
> >
> >
> >You'll have to ask Flavio because I don't really understand what he's
> >saying there, tbh. You have to have (n+1)/2 nodes available and
> >communicating to each other to maintain a live quorum (so, in your case,
> >you must have 4 nodes available at all times to maintain a live quorum).
> >
> >
> >On Tue, Oct 21, 2014 at 5:08 PM, Denis Samoilov <sa...@gmail.com>
> wrote:
> >
> >> Camille, thank you very much! Very interesting read. I also found a
> thread
> >> where you participated three years ago and found one particular comment
> a
> >> bit confusing *"One quick comment. We do not require majority quorums in
> >> ZooKeeper, and one reason we implemented this feature was exactly to
> >> enable more flexibility in deployments with multiple data centers*" (
> >>
> >>
> http://mail-archives.apache.org/mod_mbox/zookeeper-user/201109.mbox/%3C0B4CC52A-939E-4896-A269-50DC31E20AA6@yahoo-inc.com%3E
> >> )
> >> but this potentially contradicts FAQ: *"if the leader is in the
> non-quorum
> >> side of the partition, that side of the partition will recognize that
> it no
> >> longer has a quorum of the ensemble"* (
> >> https://cwiki.apache.org/confluence/display/ZOOKEEPER/FailureScenarios
> ).
> >>
> >> Where is the truth? :)
> >>
> >> On Tue, Oct 21, 2014 at 12:35 PM, Camille Fournier <ca...@apache.org>
> >> wrote:
> >>
> >> > I have a blog post on this topic:
> >> >
> >> >
> >>
> http://whilefalse.blogspot.com/2012/12/building-global-highly-available.html
> >> >
> >> > I think you will find it helpful.
> >> > The short answer is: the scheme you have proposed will cause the ZK
> to be
> >> > unavailable when you do maintenance on the data center with 4 quorum
> >> > members.
> >> >
> >> > Best,
> >> > C
> >> >
> >> > On Tue, Oct 21, 2014 at 3:03 PM, Denis Samoilov <sa...@gmail.com>
> >> > wrote:
> >> >
> >> > > hi,
> >> > >
> >> > > Could you please help to understand the following setup: we have two
> >> > > datacenters and want to setup ZK cluster so it will use servers (ZK
> >> > servers
> >> > > not clients) in both: like 3 ZK servers in DC1 and 4 ZK servers in
> DC2.
> >> > We
> >> > > sometime do maintenance in one or other DC. So ZK will completely
> lose
> >> > > replicas in one of the DC for several hours. E.g. if DC2 is under
> >> > > maintenance ZK will have only 3 out of 7 nodes and these 3 nodes
> >> supposed
> >> > > to receive writes.
> >> > >
> >> > > The questions:
> >> > > 1) is it Ok for ZK to have such setup?
> >> > > 2) will ZK catch up after losing 4 Servers and getting them back in
> >> some
> >> > > time? (this will be a majority actually :) )
> >> > > 3) what is right number of nodes, is 5 sufficient : 2 + 3?
> >> > >
> >> > > Latency between DCs is pretty low (DCs are close to each other).
> >> > >
> >> > >
> >> > > Thank you for any advice.
> >> > >
> >> > > -Denis
> >> > >
> >> >
> >>
> >
> >
> >
>
Re: cross DC setup - is it Ok for ZK?
Posted by Flavio Junqueira <fp...@yahoo.com.INVALID>.
Hierarchical quorums don't rely on strict majorities. Quorums are formed by taking majorities from a majority of groups, so if you have 3 groups of 3 servers, you have a quorum with 4 servers, 2 from each of 2 distinct groups. This is a different way of doing quorums in ZooKeeper if grouping makes sense in your scenario, like when you have multiple colos.
-Flavio
On Tuesday, October 21, 2014 10:15 PM, Camille Fournier <ca...@apache.org> wrote:
>
>
>You'll have to ask Flavio because I don't really understand what he's
>saying there, tbh. You have to have (n+1)/2 nodes available and
>communicating to each other to maintain a live quorum (so, in your case,
>you must have 4 nodes available at all times to maintain a live quorum).
>
>
>On Tue, Oct 21, 2014 at 5:08 PM, Denis Samoilov <sa...@gmail.com> wrote:
>
>> Camille, thank you very much! Very interesting read. I also found a thread
>> where you participated three years ago and found one particular comment a
>> bit confusing *"One quick comment. We do not require majority quorums in
>> ZooKeeper, and one reason we implemented this feature was exactly to
>> enable more flexibility in deployments with multiple data centers*" (
>>
>> http://mail-archives.apache.org/mod_mbox/zookeeper-user/201109.mbox/%3C0B4CC52A-939E-4896-A269-50DC31E20AA6@yahoo-inc.com%3E
>> )
>> but this potentially contradicts FAQ: *"if the leader is in the non-quorum
>> side of the partition, that side of the partition will recognize that it no
>> longer has a quorum of the ensemble"* (
>> https://cwiki.apache.org/confluence/display/ZOOKEEPER/FailureScenarios).
>>
>> Where is the truth? :)
>>
>> On Tue, Oct 21, 2014 at 12:35 PM, Camille Fournier <ca...@apache.org>
>> wrote:
>>
>> > I have a blog post on this topic:
>> >
>> >
>> http://whilefalse.blogspot.com/2012/12/building-global-highly-available.html
>> >
>> > I think you will find it helpful.
>> > The short answer is: the scheme you have proposed will cause the ZK to be
>> > unavailable when you do maintenance on the data center with 4 quorum
>> > members.
>> >
>> > Best,
>> > C
>> >
>> > On Tue, Oct 21, 2014 at 3:03 PM, Denis Samoilov <sa...@gmail.com>
>> > wrote:
>> >
>> > > hi,
>> > >
>> > > Could you please help to understand the following setup: we have two
>> > > datacenters and want to setup ZK cluster so it will use servers (ZK
>> > servers
>> > > not clients) in both: like 3 ZK servers in DC1 and 4 ZK servers in DC2.
>> > We
>> > > sometime do maintenance in one or other DC. So ZK will completely lose
>> > > replicas in one of the DC for several hours. E.g. if DC2 is under
>> > > maintenance ZK will have only 3 out of 7 nodes and these 3 nodes
>> supposed
>> > > to receive writes.
>> > >
>> > > The questions:
>> > > 1) is it Ok for ZK to have such setup?
>> > > 2) will ZK catch up after losing 4 Servers and getting them back in
>> some
>> > > time? (this will be a majority actually :) )
>> > > 3) what is right number of nodes, is 5 sufficient : 2 + 3?
>> > >
>> > > Latency between DCs is pretty low (DCs are close to each other).
>> > >
>> > >
>> > > Thank you for any advice.
>> > >
>> > > -Denis
>> > >
>> >
>>
>
>
>
Re: cross DC setup - is it Ok for ZK?
Posted by Camille Fournier <ca...@apache.org>.
You'll have to ask Flavio because I don't really understand what he's
saying there, tbh. You have to have (n+1)/2 nodes available and
communicating to each other to maintain a live quorum (so, in your case,
you must have 4 nodes available at all times to maintain a live quorum).
On Tue, Oct 21, 2014 at 5:08 PM, Denis Samoilov <sa...@gmail.com> wrote:
> Camille, thank you very much! Very interesting read. I also found a thread
> where you participated three years ago and found one particular comment a
> bit confusing *"One quick comment. We do not require majority quorums in
> ZooKeeper, and one reason we implemented this feature was exactly to
> enable more flexibility in deployments with multiple data centers*" (
>
> http://mail-archives.apache.org/mod_mbox/zookeeper-user/201109.mbox/%3C0B4CC52A-939E-4896-A269-50DC31E20AA6@yahoo-inc.com%3E
> )
> but this potentially contradicts FAQ: *"if the leader is in the non-quorum
> side of the partition, that side of the partition will recognize that it no
> longer has a quorum of the ensemble"* (
> https://cwiki.apache.org/confluence/display/ZOOKEEPER/FailureScenarios).
>
> Where is the truth? :)
>
> On Tue, Oct 21, 2014 at 12:35 PM, Camille Fournier <ca...@apache.org>
> wrote:
>
> > I have a blog post on this topic:
> >
> >
> http://whilefalse.blogspot.com/2012/12/building-global-highly-available.html
> >
> > I think you will find it helpful.
> > The short answer is: the scheme you have proposed will cause the ZK to be
> > unavailable when you do maintenance on the data center with 4 quorum
> > members.
> >
> > Best,
> > C
> >
> > On Tue, Oct 21, 2014 at 3:03 PM, Denis Samoilov <sa...@gmail.com>
> > wrote:
> >
> > > hi,
> > >
> > > Could you please help to understand the following setup: we have two
> > > datacenters and want to setup ZK cluster so it will use servers (ZK
> > servers
> > > not clients) in both: like 3 ZK servers in DC1 and 4 ZK servers in DC2.
> > We
> > > sometime do maintenance in one or other DC. So ZK will completely lose
> > > replicas in one of the DC for several hours. E.g. if DC2 is under
> > > maintenance ZK will have only 3 out of 7 nodes and these 3 nodes
> supposed
> > > to receive writes.
> > >
> > > The questions:
> > > 1) is it Ok for ZK to have such setup?
> > > 2) will ZK catch up after losing 4 Servers and getting them back in
> some
> > > time? (this will be a majority actually :) )
> > > 3) what is right number of nodes, is 5 sufficient : 2 + 3?
> > >
> > > Latency between DCs is pretty low (DCs are close to each other).
> > >
> > >
> > > Thank you for any advice.
> > >
> > > -Denis
> > >
> >
>
Re: cross DC setup - is it Ok for ZK?
Posted by Denis Samoilov <sa...@gmail.com>.
Camille, thank you very much! Very interesting read. I also found a thread
where you participated three years ago and found one particular comment a
bit confusing *"One quick comment. We do not require majority quorums in
ZooKeeper, and one reason we implemented this feature was exactly to
enable more flexibility in deployments with multiple data centers*" (
http://mail-archives.apache.org/mod_mbox/zookeeper-user/201109.mbox/%3C0B4CC52A-939E-4896-A269-50DC31E20AA6@yahoo-inc.com%3E
)
but this potentially contradicts FAQ: *"if the leader is in the non-quorum
side of the partition, that side of the partition will recognize that it no
longer has a quorum of the ensemble"* (
https://cwiki.apache.org/confluence/display/ZOOKEEPER/FailureScenarios).
Where is the truth? :)
On Tue, Oct 21, 2014 at 12:35 PM, Camille Fournier <ca...@apache.org>
wrote:
> I have a blog post on this topic:
>
> http://whilefalse.blogspot.com/2012/12/building-global-highly-available.html
>
> I think you will find it helpful.
> The short answer is: the scheme you have proposed will cause the ZK to be
> unavailable when you do maintenance on the data center with 4 quorum
> members.
>
> Best,
> C
>
> On Tue, Oct 21, 2014 at 3:03 PM, Denis Samoilov <sa...@gmail.com>
> wrote:
>
> > hi,
> >
> > Could you please help to understand the following setup: we have two
> > datacenters and want to setup ZK cluster so it will use servers (ZK
> servers
> > not clients) in both: like 3 ZK servers in DC1 and 4 ZK servers in DC2.
> We
> > sometime do maintenance in one or other DC. So ZK will completely lose
> > replicas in one of the DC for several hours. E.g. if DC2 is under
> > maintenance ZK will have only 3 out of 7 nodes and these 3 nodes supposed
> > to receive writes.
> >
> > The questions:
> > 1) is it Ok for ZK to have such setup?
> > 2) will ZK catch up after losing 4 Servers and getting them back in some
> > time? (this will be a majority actually :) )
> > 3) what is right number of nodes, is 5 sufficient : 2 + 3?
> >
> > Latency between DCs is pretty low (DCs are close to each other).
> >
> >
> > Thank you for any advice.
> >
> > -Denis
> >
>
Re: cross DC setup - is it Ok for ZK?
Posted by Camille Fournier <ca...@apache.org>.
I have a blog post on this topic:
http://whilefalse.blogspot.com/2012/12/building-global-highly-available.html
I think you will find it helpful.
The short answer is: the scheme you have proposed will cause the ZK to be
unavailable when you do maintenance on the data center with 4 quorum
members.
Best,
C
On Tue, Oct 21, 2014 at 3:03 PM, Denis Samoilov <sa...@gmail.com> wrote:
> hi,
>
> Could you please help to understand the following setup: we have two
> datacenters and want to setup ZK cluster so it will use servers (ZK servers
> not clients) in both: like 3 ZK servers in DC1 and 4 ZK servers in DC2. We
> sometime do maintenance in one or other DC. So ZK will completely lose
> replicas in one of the DC for several hours. E.g. if DC2 is under
> maintenance ZK will have only 3 out of 7 nodes and these 3 nodes supposed
> to receive writes.
>
> The questions:
> 1) is it Ok for ZK to have such setup?
> 2) will ZK catch up after losing 4 Servers and getting them back in some
> time? (this will be a majority actually :) )
> 3) what is right number of nodes, is 5 sufficient : 2 + 3?
>
> Latency between DCs is pretty low (DCs are close to each other).
>
>
> Thank you for any advice.
>
> -Denis
>