Zookeeper Cross Datacenter Cluster

[Le Cyberian <le...@gmail.com>]

Hi Guys,

I would like to have Kafka Cluster which is dependent on Zookeeper like
many other great projects out their. Kafka would have 4 nodes as its
cluster since their having even number is not a problem because it does
leader election based its config it keeps in zookeeper.

My scenario is having two server rooms on different floors having fiber
connectivity with each other so network latency / lag is almost like a
local connection.

However, I would like to implement zookeeper cluster between both of them
for auto-failover / redundancy purposes. So if one server room is down
other should work properly.

For example: on a 5 node zookeeper cluster distributed on both of the
server rooms with 3 in one and 2 in other would give me 2 node failure
tolerance, However if the server room which has 3 servers is down then
ensemble would be broken and split brain.

Can you please suggest how to achieve this to have failover / redundancy
between two server rooms or two locations lets say ?

Is it possible to run 3 node cluster in each server room and have some sort
of master-master between them ?

Thank for your time and helping out in advance.

Kind regards,

Le

Re: Zookeeper Cross Datacenter Cluster

[Jordan Zimmerman <jo...@jordanzimmerman.com>]

Consider using 3 DCs, not 2. If you have 3 then you can stand to lose 1 of them. Think of each data center as an instance in a quorum. Always use odd numbers when possible.

-JZ

> On Mar 6, 2017, at 2:42 PM, Le Cyberian <le...@gmail.com> wrote:
> 
> Thank for your replies.
> 
> I see, in that way then failover won't be possible as if the main/major DC
> is down.
> 
> Is there something i can do using hierarchical quorum groups or observers ?
> 
> I am quiet confused to achieve this setup somehow, as how someone would
> build a failover over 2 dc's :-/
> 
> On Mon, Mar 6, 2017 at 11:08 PM, Van Klaveren, Brian N. <
> bvan@slac.stanford.edu> wrote:
> 
>> Or, an extra server in a third datacenter to break the tie.
>> 
>> Brian
>> 
>> 
>>> On Mar 6, 2017, at 2:05 PM, Jordan Zimmerman <jo...@jordanzimmerman.com>
>> wrote:
>>> 
>>> This scenario is not possible. An even number of servers doesn't help
>> you as a quorum in ZooKeeper is (N/2)+1. So, if you put 2 servers in each
>> DC, a network partition would disable BOTH DCs. (4/2)+1 == 3. So, the only
>> option is to choose one of the DCs as the main DC and put an extra server
>> there. This way you could lose the "minor" DC.
>>> 
>>> -Jordan
>>> 
>>>> On Mar 6, 2017, at 11:34 AM, Le Cyberian <le...@gmail.com> wrote:
>>>> 
>>>> Hi Guys,
>>>> 
>>>> I would like to have Kafka Cluster which is dependent on Zookeeper like
>>>> many other great projects out their. Kafka would have 4 nodes as its
>>>> cluster since their having even number is not a problem because it does
>>>> leader election based its config it keeps in zookeeper.
>>>> 
>>>> My scenario is having two server rooms on different floors having fiber
>>>> connectivity with each other so network latency / lag is almost like a
>>>> local connection.
>>>> 
>>>> However, I would like to implement zookeeper cluster between both of
>> them
>>>> for auto-failover / redundancy purposes. So if one server room is down
>>>> other should work properly.
>>>> 
>>>> For example: on a 5 node zookeeper cluster distributed on both of the
>>>> server rooms with 3 in one and 2 in other would give me 2 node failure
>>>> tolerance, However if the server room which has 3 servers is down then
>>>> ensemble would be broken and split brain.
>>>> 
>>>> Can you please suggest how to achieve this to have failover / redundancy
>>>> between two server rooms or two locations lets say ?
>>>> 
>>>> Is it possible to run 3 node cluster in each server room and have some
>> sort
>>>> of master-master between them ?
>>>> 
>>>> Thank for your time and helping out in advance.
>>>> 
>>>> Kind regards,
>>>> 
>>>> Le
>>> 
>> 
>> 

Re: Zookeeper Cross Datacenter Cluster

[Michael Han <ha...@cloudera.com>]

Back up requires replication which has two types, synchronous and
asynchronous.

ZooKeeper quorum provides synchronous replication. But as mentioned, 2 DC
will not work no matter how. You need at least three (and in general odd
numbers - for majority quorum). There are quorum weights and groups that
you might leverage but those would not solve 2 DC fault tolerant case.
There were a good discussion awhile ago about these concepts and multi-DC
set up for reference. [1]

ZooKeeper observer provides asynchronous replication. One setup is to have
a quorum of servers in one DC and having all observers in another. As with
any async replication, you might lose data, and whether or not this is OK
depends on your use case and your requirement on data integrity.

For fail over, using quorum you get it for free, as no extra work is
required. Using observer to fail over requires to reconfigure and restart
the ensemble by promoting observers to followers, so it is not
automatically provided by ZK and you'd need application logic to handle it
or do it manually.

There are other "external" approaches of backing up for example you can run
a background job to periodically backup ZK transaction log and snapshots
and store those somewhere then later spin a new ensemble and recover from
those. This approach is likely to lose data as well as its async
replication and it also requires application logic to handle the recover
process or do it manually...

[1]
http://zookeeper-user.578899.n2.nabble.com/zookeeper-deployment-strategy-for-multi-data-centers-td7582358.html

On Mon, Mar 6, 2017 at 2:42 PM, Le Cyberian <le...@gmail.com> wrote:

> Thank for your replies.
>
> I see, in that way then failover won't be possible as if the main/major DC
> is down.
>
> Is there something i can do using hierarchical quorum groups or observers ?
>
> I am quiet confused to achieve this setup somehow, as how someone would
> build a failover over 2 dc's :-/
>
> On Mon, Mar 6, 2017 at 11:08 PM, Van Klaveren, Brian N. <
> bvan@slac.stanford.edu> wrote:
>
> > Or, an extra server in a third datacenter to break the tie.
> >
> > Brian
> >
> >
> > > On Mar 6, 2017, at 2:05 PM, Jordan Zimmerman <
> jordan@jordanzimmerman.com>
> > wrote:
> > >
> > > This scenario is not possible. An even number of servers doesn't help
> > you as a quorum in ZooKeeper is (N/2)+1. So, if you put 2 servers in each
> > DC, a network partition would disable BOTH DCs. (4/2)+1 == 3. So, the
> only
> > option is to choose one of the DCs as the main DC and put an extra server
> > there. This way you could lose the "minor" DC.
> > >
> > > -Jordan
> > >
> > >> On Mar 6, 2017, at 11:34 AM, Le Cyberian <le...@gmail.com>
> wrote:
> > >>
> > >> Hi Guys,
> > >>
> > >> I would like to have Kafka Cluster which is dependent on Zookeeper
> like
> > >> many other great projects out their. Kafka would have 4 nodes as its
> > >> cluster since their having even number is not a problem because it
> does
> > >> leader election based its config it keeps in zookeeper.
> > >>
> > >> My scenario is having two server rooms on different floors having
> fiber
> > >> connectivity with each other so network latency / lag is almost like a
> > >> local connection.
> > >>
> > >> However, I would like to implement zookeeper cluster between both of
> > them
> > >> for auto-failover / redundancy purposes. So if one server room is down
> > >> other should work properly.
> > >>
> > >> For example: on a 5 node zookeeper cluster distributed on both of the
> > >> server rooms with 3 in one and 2 in other would give me 2 node failure
> > >> tolerance, However if the server room which has 3 servers is down then
> > >> ensemble would be broken and split brain.
> > >>
> > >> Can you please suggest how to achieve this to have failover /
> redundancy
> > >> between two server rooms or two locations lets say ?
> > >>
> > >> Is it possible to run 3 node cluster in each server room and have some
> > sort
> > >> of master-master between them ?
> > >>
> > >> Thank for your time and helping out in advance.
> > >>
> > >> Kind regards,
> > >>
> > >> Le
> > >
> >
> >
>



-- 
Cheers
Michael.

Re: Zookeeper Cross Datacenter Cluster

[Le Cyberian <le...@gmail.com>]

Thank for your replies.

I see, in that way then failover won't be possible as if the main/major DC
is down.

Is there something i can do using hierarchical quorum groups or observers ?

I am quiet confused to achieve this setup somehow, as how someone would
build a failover over 2 dc's :-/

On Mon, Mar 6, 2017 at 11:08 PM, Van Klaveren, Brian N. <
bvan@slac.stanford.edu> wrote:

> Or, an extra server in a third datacenter to break the tie.
>
> Brian
>
>
> > On Mar 6, 2017, at 2:05 PM, Jordan Zimmerman <jo...@jordanzimmerman.com>
> wrote:
> >
> > This scenario is not possible. An even number of servers doesn't help
> you as a quorum in ZooKeeper is (N/2)+1. So, if you put 2 servers in each
> DC, a network partition would disable BOTH DCs. (4/2)+1 == 3. So, the only
> option is to choose one of the DCs as the main DC and put an extra server
> there. This way you could lose the "minor" DC.
> >
> > -Jordan
> >
> >> On Mar 6, 2017, at 11:34 AM, Le Cyberian <le...@gmail.com> wrote:
> >>
> >> Hi Guys,
> >>
> >> I would like to have Kafka Cluster which is dependent on Zookeeper like
> >> many other great projects out their. Kafka would have 4 nodes as its
> >> cluster since their having even number is not a problem because it does
> >> leader election based its config it keeps in zookeeper.
> >>
> >> My scenario is having two server rooms on different floors having fiber
> >> connectivity with each other so network latency / lag is almost like a
> >> local connection.
> >>
> >> However, I would like to implement zookeeper cluster between both of
> them
> >> for auto-failover / redundancy purposes. So if one server room is down
> >> other should work properly.
> >>
> >> For example: on a 5 node zookeeper cluster distributed on both of the
> >> server rooms with 3 in one and 2 in other would give me 2 node failure
> >> tolerance, However if the server room which has 3 servers is down then
> >> ensemble would be broken and split brain.
> >>
> >> Can you please suggest how to achieve this to have failover / redundancy
> >> between two server rooms or two locations lets say ?
> >>
> >> Is it possible to run 3 node cluster in each server room and have some
> sort
> >> of master-master between them ?
> >>
> >> Thank for your time and helping out in advance.
> >>
> >> Kind regards,
> >>
> >> Le
> >
>
>

Re: Zookeeper Cross Datacenter Cluster

["Van Klaveren, Brian N." <bv...@slac.stanford.edu>]

Or, an extra server in a third datacenter to break the tie.

Brian


> On Mar 6, 2017, at 2:05 PM, Jordan Zimmerman <jo...@jordanzimmerman.com> wrote:
> 
> This scenario is not possible. An even number of servers doesn't help you as a quorum in ZooKeeper is (N/2)+1. So, if you put 2 servers in each DC, a network partition would disable BOTH DCs. (4/2)+1 == 3. So, the only option is to choose one of the DCs as the main DC and put an extra server there. This way you could lose the "minor" DC.
> 
> -Jordan
> 
>> On Mar 6, 2017, at 11:34 AM, Le Cyberian <le...@gmail.com> wrote:
>> 
>> Hi Guys,
>> 
>> I would like to have Kafka Cluster which is dependent on Zookeeper like
>> many other great projects out their. Kafka would have 4 nodes as its
>> cluster since their having even number is not a problem because it does
>> leader election based its config it keeps in zookeeper.
>> 
>> My scenario is having two server rooms on different floors having fiber
>> connectivity with each other so network latency / lag is almost like a
>> local connection.
>> 
>> However, I would like to implement zookeeper cluster between both of them
>> for auto-failover / redundancy purposes. So if one server room is down
>> other should work properly.
>> 
>> For example: on a 5 node zookeeper cluster distributed on both of the
>> server rooms with 3 in one and 2 in other would give me 2 node failure
>> tolerance, However if the server room which has 3 servers is down then
>> ensemble would be broken and split brain.
>> 
>> Can you please suggest how to achieve this to have failover / redundancy
>> between two server rooms or two locations lets say ?
>> 
>> Is it possible to run 3 node cluster in each server room and have some sort
>> of master-master between them ?
>> 
>> Thank for your time and helping out in advance.
>> 
>> Kind regards,
>> 
>> Le
> 

Re: Zookeeper Cross Datacenter Cluster

[Jordan Zimmerman <jo...@jordanzimmerman.com>]

This scenario is not possible. An even number of servers doesn't help you as a quorum in ZooKeeper is (N/2)+1. So, if you put 2 servers in each DC, a network partition would disable BOTH DCs. (4/2)+1 == 3. So, the only option is to choose one of the DCs as the main DC and put an extra server there. This way you could lose the "minor" DC.

-Jordan

> On Mar 6, 2017, at 11:34 AM, Le Cyberian <le...@gmail.com> wrote:
> 
> Hi Guys,
> 
> I would like to have Kafka Cluster which is dependent on Zookeeper like
> many other great projects out their. Kafka would have 4 nodes as its
> cluster since their having even number is not a problem because it does
> leader election based its config it keeps in zookeeper.
> 
> My scenario is having two server rooms on different floors having fiber
> connectivity with each other so network latency / lag is almost like a
> local connection.
> 
> However, I would like to implement zookeeper cluster between both of them
> for auto-failover / redundancy purposes. So if one server room is down
> other should work properly.
> 
> For example: on a 5 node zookeeper cluster distributed on both of the
> server rooms with 3 in one and 2 in other would give me 2 node failure
> tolerance, However if the server room which has 3 servers is down then
> ensemble would be broken and split brain.
> 
> Can you please suggest how to achieve this to have failover / redundancy
> between two server rooms or two locations lets say ?
> 
> Is it possible to run 3 node cluster in each server room and have some sort
> of master-master between them ?
> 
> Thank for your time and helping out in advance.
> 
> Kind regards,
> 
> Le