Allowing a ZooKeeper server to be part of multiple clusters

[Vishal K <vi...@gmail.com>]

Hi All,

I am thinking about the choices one would have to support multiple 2-node
clusters. Assume that for some reason one needs to support multiple 2-node
clusters.
This would mean they will have to figure out a way to run a third instance
of ZK server for each cluster somewhere to ensure that a ZK cluster is
available after a failure.

This works well if we have to run one or two 2-node clusters. However, what
if we have to run many 2-node clusters?

I have following options:
1. Find m machines to run the third instance of each cluster. Run n/m
instances of ZK on each machine.
2. Modify ZooKeeper server to participate in multiple clusters. This will
allow us to run y instances of third node where each instance will be part
of n/y clusters.
3. Run the third instance of ZK server required for the ith cluster on one
of the server on (i+1)%n cluster. Essentially, distribute the third instance
across the other clusters.

The pros and cons of each approach are fairly obvious. While I prefer the
third approach, I would like to check what everyone thinks about the second
approach.

Thanks.
-Vishal

Re: Allowing a ZooKeeper server to be part of multiple clusters

[Flavio Junqueira <fp...@yahoo-inc.com>]

That proposal came in the context of federated zookeeper, and the  
motivation at the time was to use multiple overlapping clusters to  
enable increasing write throughput as we increase the number of  
servers. To my knowledge, we haven't made any progress on the  
implementation of such a feature.

I'd be curious to understand what scenario Vishal envision for such a  
2-node cluster feature. If it is not federated, then we would have  
trouble with ZooKeeper because we rely upon one single leader to  
generate state updates. In the federated case, there is one leader  
(perhaps multiple during non-overlapping periods of time) for each
partition.

There is this wiki page I have written a while back:

	http://wiki.apache.org/hadoop/ZooKeeper/PartitionedZookeeper

Hope it helps.

-Flavio

On Oct 25, 2010, at 11:24 PM, Vishal K wrote:

> Hi Mahadev,
>
> It lets one run multiple 2-node clusters. Suppose I have an  
> application that
> does a simple 2-way mirroring of my data and uses ZK for clustering.  
> If I
> need to support many 2-node clusters, where will I find the spare  
> machines
> to run the third instance for each cluster?
>
> -Vishal
>
> On Mon, Oct 25, 2010 at 5:14 PM, Mahadev Konar <mahadev@yahoo- 
> inc.com>wrote:
>
>> Hi Vishal,
>> This idea  (2.) had been kicked around intially by Flavio. I think  
>> he¹ll
>> probably chip in on the discussion. I am just curious on the whats  
>> the idea
>> behind your proposal? Is this to provide some kind of failure  
>> gaurantees
>> between a 2 node and 3 node cluster?
>>
>> Thanks
>> mahadev
>>
>> On 10/25/10 1:05 PM, "Vishal K" <vi...@gmail.com> wrote:
>>
>>> Hi All,
>>>
>>> I am thinking about the choices one would have to support multiple  
>>> 2-node
>>> clusters. Assume that for some reason one needs to support multiple
>> 2-node
>>> clusters.
>>> This would mean they will have to figure out a way to run a third
>> instance
>>> of ZK server for each cluster somewhere to ensure that a ZK  
>>> cluster is
>>> available after a failure.
>>>
>>> This works well if we have to run one or two 2-node clusters.  
>>> However,
>> what
>>> if we have to run many 2-node clusters?
>>>
>>> I have following options:
>>> 1. Find m machines to run the third instance of each cluster. Run  
>>> n/m
>>> instances of ZK on each machine.
>>> 2. Modify ZooKeeper server to participate in multiple clusters.  
>>> This will
>>> allow us to run y instances of third node where each instance will  
>>> be
>> part
>>> of n/y clusters.
>>> 3. Run the third instance of ZK server required for the ith  
>>> cluster on
>> one
>>> of the server on (i+1)%n cluster. Essentially, distribute the third
>> instance
>>> across the other clusters.
>>>
>>> The pros and cons of each approach are fairly obvious. While I  
>>> prefer the
>>> third approach, I would like to check what everyone thinks about the
>> second
>>> approach.
>>>
>>> Thanks.
>>> -Vishal
>>>
>>
>>

flavio
junqueira

research scientist

fpj@yahoo-inc.com
direct +34 93-183-8828

avinguda diagonal 177, 8th floor, barcelona, 08018, es
phone (408) 349 3300    fax (408) 349 3301

Re: Allowing a ZooKeeper server to be part of multiple clusters

[Vishal K <vi...@gmail.com>]

Hi Mahadev,

It lets one run multiple 2-node clusters. Suppose I have an application that
does a simple 2-way mirroring of my data and uses ZK for clustering. If I
need to support many 2-node clusters, where will I find the spare machines
to run the third instance for each cluster?

-Vishal

On Mon, Oct 25, 2010 at 5:14 PM, Mahadev Konar <ma...@yahoo-inc.com>wrote:

> Hi Vishal,
>  This idea  (2.) had been kicked around intially by Flavio. I think he¹ll
> probably chip in on the discussion. I am just curious on the whats the idea
> behind your proposal? Is this to provide some kind of failure gaurantees
> between a 2 node and 3 node cluster?
>
> Thanks
> mahadev
>
> On 10/25/10 1:05 PM, "Vishal K" <vi...@gmail.com> wrote:
>
> > Hi All,
> >
> > I am thinking about the choices one would have to support multiple 2-node
> > clusters. Assume that for some reason one needs to support multiple
> 2-node
> > clusters.
> > This would mean they will have to figure out a way to run a third
> instance
> > of ZK server for each cluster somewhere to ensure that a ZK cluster is
> > available after a failure.
> >
> > This works well if we have to run one or two 2-node clusters. However,
> what
> > if we have to run many 2-node clusters?
> >
> > I have following options:
> > 1. Find m machines to run the third instance of each cluster. Run n/m
> > instances of ZK on each machine.
> > 2. Modify ZooKeeper server to participate in multiple clusters. This will
> > allow us to run y instances of third node where each instance will be
> part
> > of n/y clusters.
> > 3. Run the third instance of ZK server required for the ith cluster on
> one
> > of the server on (i+1)%n cluster. Essentially, distribute the third
> instance
> > across the other clusters.
> >
> > The pros and cons of each approach are fairly obvious. While I prefer the
> > third approach, I would like to check what everyone thinks about the
> second
> > approach.
> >
> > Thanks.
> > -Vishal
> >
>
>

Re: Allowing a ZooKeeper server to be part of multiple clusters

[Mahadev Konar <ma...@yahoo-inc.com>]

Hi Vishal,
 This idea  (2.) had been kicked around intially by Flavio. I think he¹ll
probably chip in on the discussion. I am just curious on the whats the idea
behind your proposal? Is this to provide some kind of failure gaurantees
between a 2 node and 3 node cluster?

Thanks
mahadev

On 10/25/10 1:05 PM, "Vishal K" <vi...@gmail.com> wrote:

> Hi All,
> 
> I am thinking about the choices one would have to support multiple 2-node
> clusters. Assume that for some reason one needs to support multiple 2-node
> clusters.
> This would mean they will have to figure out a way to run a third instance
> of ZK server for each cluster somewhere to ensure that a ZK cluster is
> available after a failure.
> 
> This works well if we have to run one or two 2-node clusters. However, what
> if we have to run many 2-node clusters?
> 
> I have following options:
> 1. Find m machines to run the third instance of each cluster. Run n/m
> instances of ZK on each machine.
> 2. Modify ZooKeeper server to participate in multiple clusters. This will
> allow us to run y instances of third node where each instance will be part
> of n/y clusters.
> 3. Run the third instance of ZK server required for the ith cluster on one
> of the server on (i+1)%n cluster. Essentially, distribute the third instance
> across the other clusters.
> 
> The pros and cons of each approach are fairly obvious. While I prefer the
> third approach, I would like to check what everyone thinks about the second
> approach.
> 
> Thanks.
> -Vishal
>