Deployment planning question

[Jason Harmon <jh...@att.com>]

We are planning a ZooKeeper deployment, but are struggling with some aspects of our current architecture. We have two data centers; most applications use load balancing between the two data centers to ensure redundancy for disaster recovery etc. ZooKeeper has been challenging for us, because we can't quite figure out how to structure our servers. We aren't starting with lots of clients, so 3 servers should be a good fit initially. However, if we put two servers in one datacenter, and one server in the other, we're setup for a quorum failure if we have a power outage at the datacenter with two boxes.
Short of setting up a third data center for our division, what options do we have for deploying ZooKeeper effectively, with disaster recovery in mind etc.
Thanks!

Jason Harmon
Senior Software Architect

RE: Deployment planning question

[Jason Harmon <jh...@att.com>]

So I tried out the 'mirrored' option last night. I setup one box as a standalone, with the intention of an observer as a second box. Unfortunately it complained that there was no quorum. In looking through the ZK code, it's apparent it expects to support observers only when there is an active ensemble (3 or more servers). 

Here's the thing: I'm only looking to use leader election for an app at this point, with some potential to store configuration data and some small coordination data between a few applications. Our IT requirements demand I have total redundancy in two datacenters for disaster recovery planning. The more I look at this, the more I think ZooKeeper is beautifully elegant in terms of code/usage, but the deployment aspect just isn't fitting.

-----Original Message-----
From: Ted Dunning [mailto:ted.dunning@gmail.com] 
Sent: Saturday, February 04, 2012 1:24 AM
To: user@zookeeper.apache.org
Subject: Re: Deployment planning question

> You can do that.  Just define the mirror as an observer.

<snip>

>> corporate security restrictions. It sure would be nice for a 
>> smaller-scale deployment to just have a simple mirrored option, as 
>> opposed to the mandatory 3 server/quorum requirement.
>>

Re: Deployment planning question

[Ted Dunning <te...@gmail.com>]

On Fri, Feb 3, 2012 at 4:01 PM, Jason Harmon <jh...@att.com> wrote:

> My preference, of course, would be to have three datacenters...if one is
> partitioned off, zookeeper would not respond, which would be perfect. In
> that scenario, our other two would still be up and running in the other two
> datacenters.
>

Sure.  But there is likely to be some residue of clients who are
partitioned off with the minority side of the fence.  Depending on your
application, this can often be made to work well.

Of course I only have two, and I can't host anything externally due to
> corporate security restrictions. It sure would be nice for a smaller-scale
> deployment to just have a simple mirrored option, as opposed to the
> mandatory 3 server/quorum requirement.
>

You can do that.  Just define the mirror as an observer.

Re: Deployment planning question

[Benjamin Reed <br...@apache.org>]

im curious what you want to happen in the mirrored option. if you are
mirroring and both datacenters are up, but cannot talk to each other,
what should happen?

On Fri, Feb 3, 2012 at 4:01 PM, Jason Harmon <jh...@att.com> wrote:
> My preference, of course, would be to have three datacenters...if one is partitioned off, zookeeper would not respond, which would be perfect. In that scenario, our other two would still be up and running in the other two datacenters.
> Of course I only have two, and I can't host anything externally due to corporate security restrictions. It sure would be nice for a smaller-scale deployment to just have a simple mirrored option, as opposed to the mandatory 3 server/quorum requirement.
> I'm investigating a potential third site I can get access to, but I'm just trying to fully explore any other potential options.
>
> Thanks for the quick response Ben
>
> -----Original Message-----
> From: Benjamin Reed [mailto:breed@apache.org]
> Sent: Friday, February 03, 2012 4:05 PM
> To: user@zookeeper.apache.org
> Subject: Re: Deployment planning question
>
> what do you want to happen if both datacenters are up, but there is a partition so that they cannot communicate with each other? answering that question may get you closer to an answer.
>
> i think your two main options are to:
>
> 1) designate a data center that you require to be up for things to work. then you can put two in that one and one in the other
> 2) otherwise you need to setup a machine in another datacenter or amazon instance or something that will in effect decide which data center can run if one of the datacenters goes down or you are in the partition scenario above.
>
> ben
>
> On Fri, Feb 3, 2012 at 1:01 PM, Jason Harmon <jh...@att.com> wrote:
>> We are planning a ZooKeeper deployment, but are struggling with some aspects of our current architecture. We have two data centers; most applications use load balancing between the two data centers to ensure redundancy for disaster recovery etc. ZooKeeper has been challenging for us, because we can't quite figure out how to structure our servers. We aren't starting with lots of clients, so 3 servers should be a good fit initially. However, if we put two servers in one datacenter, and one server in the other, we're setup for a quorum failure if we have a power outage at the datacenter with two boxes.
>> Short of setting up a third data center for our division, what options do we have for deploying ZooKeeper effectively, with disaster recovery in mind etc.
>> Thanks!
>>
>> Jason Harmon
>> Senior Software Architect
>>

RE: Deployment planning question

[Jason Harmon <jh...@att.com>]

My preference, of course, would be to have three datacenters...if one is partitioned off, zookeeper would not respond, which would be perfect. In that scenario, our other two would still be up and running in the other two datacenters.
Of course I only have two, and I can't host anything externally due to corporate security restrictions. It sure would be nice for a smaller-scale deployment to just have a simple mirrored option, as opposed to the mandatory 3 server/quorum requirement. 
I'm investigating a potential third site I can get access to, but I'm just trying to fully explore any other potential options.

Thanks for the quick response Ben

-----Original Message-----
From: Benjamin Reed [mailto:breed@apache.org] 
Sent: Friday, February 03, 2012 4:05 PM
To: user@zookeeper.apache.org
Subject: Re: Deployment planning question

what do you want to happen if both datacenters are up, but there is a partition so that they cannot communicate with each other? answering that question may get you closer to an answer.

i think your two main options are to:

1) designate a data center that you require to be up for things to work. then you can put two in that one and one in the other
2) otherwise you need to setup a machine in another datacenter or amazon instance or something that will in effect decide which data center can run if one of the datacenters goes down or you are in the partition scenario above.

ben

On Fri, Feb 3, 2012 at 1:01 PM, Jason Harmon <jh...@att.com> wrote:
> We are planning a ZooKeeper deployment, but are struggling with some aspects of our current architecture. We have two data centers; most applications use load balancing between the two data centers to ensure redundancy for disaster recovery etc. ZooKeeper has been challenging for us, because we can't quite figure out how to structure our servers. We aren't starting with lots of clients, so 3 servers should be a good fit initially. However, if we put two servers in one datacenter, and one server in the other, we're setup for a quorum failure if we have a power outage at the datacenter with two boxes.
> Short of setting up a third data center for our division, what options do we have for deploying ZooKeeper effectively, with disaster recovery in mind etc.
> Thanks!
>
> Jason Harmon
> Senior Software Architect
>

RE: Deployment planning question

[Jason Harmon <jh...@att.com>]

This is a very interesting design I had not considered. I'm going to discuss this with the team on Monday and see how it might fit our requirements.
Thanks a bunch for all the input, this is very helpful.

-----Original Message-----
From: Benjamin Reed [mailto:breed@apache.org] 
Sent: Friday, February 03, 2012 6:20 PM
To: user@zookeeper.apache.org
Subject: Re: Deployment planning question

yes this is correct. we have sites that do cross data center coordination by basically partitioning the data. the coordination data that pertains to data center 1 will have a set of machines all in data center 1 and perhaps an observer in data center 2, but all changes to that data will be done exclusively in data center 1. then another set of machines is setup in data center 2, with an observer or two in data center 1. if you can partition your coordination data that way, then this setup should work pretty well for you.

ben

On Fri, Feb 3, 2012 at 3:52 PM, Ted Dunning <te...@gmail.com> wrote:
> Another option is to design for partition by putting a ZK in each 
> datacenter.  There are a variety of ways to detect partition so that 
> each datacenter can function, but also know if there is a partition happening.
>  It is often true that useful things can be done even in the case of 
> partition.
>
> On Fri, Feb 3, 2012 at 2:05 PM, Benjamin Reed <br...@apache.org> wrote:
>
>> what do you want to happen if both datacenters are up, but there is a 
>> partition so that they cannot communicate with each other? answering 
>> that question may get you closer to an answer.
>>
>> i think your two main options are to:
>>
>> 1) designate a data center that you require to be up for things to 
>> work. then you can put two in that one and one in the other
>> 2) otherwise you need to setup a machine in another datacenter or 
>> amazon instance or something that will in effect decide which data 
>> center can run if one of the datacenters goes down or you are in the 
>> partition scenario above.
>>
>> ben
>>
>> On Fri, Feb 3, 2012 at 1:01 PM, Jason Harmon <jh...@att.com> wrote:
>> > We are planning a ZooKeeper deployment, but are struggling with 
>> > some
>> aspects of our current architecture. We have two data centers; most 
>> applications use load balancing between the two data centers to 
>> ensure redundancy for disaster recovery etc. ZooKeeper has been 
>> challenging for us, because we can't quite figure out how to 
>> structure our servers. We aren't starting with lots of clients, so 3 
>> servers should be a good fit initially. However, if we put two 
>> servers in one datacenter, and one server in the other, we're setup 
>> for a quorum failure if we have a power outage at the datacenter with two boxes.
>> > Short of setting up a third data center for our division, what 
>> > options
>> do we have for deploying ZooKeeper effectively, with disaster 
>> recovery in mind etc.
>> > Thanks!
>> >
>> > Jason Harmon
>> > Senior Software Architect
>> >
>>

Re: Deployment planning question

[Benjamin Reed <br...@apache.org>]

yes this is correct. we have sites that do cross data center
coordination by basically partitioning the data. the coordination data
that pertains to data center 1 will have a set of machines all in data
center 1 and perhaps an observer in data center 2, but all changes to
that data will be done exclusively in data center 1. then another set
of machines is setup in data center 2, with an observer or two in data
center 1. if you can partition your coordination data that way, then
this setup should work pretty well for you.

ben

On Fri, Feb 3, 2012 at 3:52 PM, Ted Dunning <te...@gmail.com> wrote:
> Another option is to design for partition by putting a ZK in each
> datacenter.  There are a variety of ways to detect partition so that each
> datacenter can function, but also know if there is a partition happening.
>  It is often true that useful things can be done even in the case of
> partition.
>
> On Fri, Feb 3, 2012 at 2:05 PM, Benjamin Reed <br...@apache.org> wrote:
>
>> what do you want to happen if both datacenters are up, but there is a
>> partition so that they cannot communicate with each other? answering
>> that question may get you closer to an answer.
>>
>> i think your two main options are to:
>>
>> 1) designate a data center that you require to be up for things to
>> work. then you can put two in that one and one in the other
>> 2) otherwise you need to setup a machine in another datacenter or
>> amazon instance or something that will in effect decide which data
>> center can run if one of the datacenters goes down or you are in the
>> partition scenario above.
>>
>> ben
>>
>> On Fri, Feb 3, 2012 at 1:01 PM, Jason Harmon <jh...@att.com> wrote:
>> > We are planning a ZooKeeper deployment, but are struggling with some
>> aspects of our current architecture. We have two data centers; most
>> applications use load balancing between the two data centers to ensure
>> redundancy for disaster recovery etc. ZooKeeper has been challenging for
>> us, because we can't quite figure out how to structure our servers. We
>> aren't starting with lots of clients, so 3 servers should be a good fit
>> initially. However, if we put two servers in one datacenter, and one server
>> in the other, we're setup for a quorum failure if we have a power outage at
>> the datacenter with two boxes.
>> > Short of setting up a third data center for our division, what options
>> do we have for deploying ZooKeeper effectively, with disaster recovery in
>> mind etc.
>> > Thanks!
>> >
>> > Jason Harmon
>> > Senior Software Architect
>> >
>>

Re: Deployment planning question

[Ted Dunning <te...@gmail.com>]

Another option is to design for partition by putting a ZK in each
datacenter.  There are a variety of ways to detect partition so that each
datacenter can function, but also know if there is a partition happening.
 It is often true that useful things can be done even in the case of
partition.

On Fri, Feb 3, 2012 at 2:05 PM, Benjamin Reed <br...@apache.org> wrote:

> what do you want to happen if both datacenters are up, but there is a
> partition so that they cannot communicate with each other? answering
> that question may get you closer to an answer.
>
> i think your two main options are to:
>
> 1) designate a data center that you require to be up for things to
> work. then you can put two in that one and one in the other
> 2) otherwise you need to setup a machine in another datacenter or
> amazon instance or something that will in effect decide which data
> center can run if one of the datacenters goes down or you are in the
> partition scenario above.
>
> ben
>
> On Fri, Feb 3, 2012 at 1:01 PM, Jason Harmon <jh...@att.com> wrote:
> > We are planning a ZooKeeper deployment, but are struggling with some
> aspects of our current architecture. We have two data centers; most
> applications use load balancing between the two data centers to ensure
> redundancy for disaster recovery etc. ZooKeeper has been challenging for
> us, because we can't quite figure out how to structure our servers. We
> aren't starting with lots of clients, so 3 servers should be a good fit
> initially. However, if we put two servers in one datacenter, and one server
> in the other, we're setup for a quorum failure if we have a power outage at
> the datacenter with two boxes.
> > Short of setting up a third data center for our division, what options
> do we have for deploying ZooKeeper effectively, with disaster recovery in
> mind etc.
> > Thanks!
> >
> > Jason Harmon
> > Senior Software Architect
> >
>

Re: Deployment planning question

[Benjamin Reed <br...@apache.org>]

what do you want to happen if both datacenters are up, but there is a
partition so that they cannot communicate with each other? answering
that question may get you closer to an answer.

i think your two main options are to:

1) designate a data center that you require to be up for things to
work. then you can put two in that one and one in the other
2) otherwise you need to setup a machine in another datacenter or
amazon instance or something that will in effect decide which data
center can run if one of the datacenters goes down or you are in the
partition scenario above.

ben

On Fri, Feb 3, 2012 at 1:01 PM, Jason Harmon <jh...@att.com> wrote:
> We are planning a ZooKeeper deployment, but are struggling with some aspects of our current architecture. We have two data centers; most applications use load balancing between the two data centers to ensure redundancy for disaster recovery etc. ZooKeeper has been challenging for us, because we can't quite figure out how to structure our servers. We aren't starting with lots of clients, so 3 servers should be a good fit initially. However, if we put two servers in one datacenter, and one server in the other, we're setup for a quorum failure if we have a power outage at the datacenter with two boxes.
> Short of setting up a third data center for our division, what options do we have for deploying ZooKeeper effectively, with disaster recovery in mind etc.
> Thanks!
>
> Jason Harmon
> Senior Software Architect
>