Understanding zookeper and kafka server failures

[Sachin Mittal <sj...@gmail.com>]

Hi,
We have following setup.
1. Three instances of zookeeper on three machines.
2. Three instances of kafka server on same three machines.
3. All the topics have replication factor 3.

So when we create a topic on any node, i see that it gets replicated on all
three instances.
I also see that topic data is getting replicated to all three nodes.

The data to main topic is written by three producers to which all three zk
nodes config is provided in connect string.

This is all working fine.

The question is what happens if one of the zookeeper crashes. Will the
broker on that node also will crash?

What happens if broker crashes? I suppose other two brokers will take the
load.

Now what happens if 2 zookeeper nodes crashes.
Or if 2 brokers crashes. Will my cluster be still working in this case.

So basically what is the difference between zookeeper failure and
server/broker failure.

Thanks
Sachin

Re: Understanding zookeper and kafka server failures

[Karolis Pocius <k....@adform.com>]

It depends on the size and load of your cluster. Zookeeper is very I/O 
sensitive, so at least you have to make sure it doesn't share disk with 
the OS or Kafka.

I assume you've read the documentation, but you might want to have a 
look at https://kafka.apache.org/documentation.html#zkops again, it 
provides reasoning behind why you shouldn't run zookeeper and kafka 
together.


On 2016.11.08 17:47, Sachin Mittal wrote:
> Hi,
> Thanks for the reply. From one obvious reason that is server crashes then
> both zookeeper and broker crashes, is there any other reason why we should
> not run broker and zookeeper on same server.
>
> If chances of server crash are extremely low can can be brought back up
> quickly, then can we keep both on same server.
>
> Thanks
> Sachin
>
>
>
> On Tue, Nov 8, 2016 at 8:59 PM, Karolis Pocius <k....@adform.com> wrote:
>
>> The question is what happens if one of the zookeeper crashes. Will the
>>> broker on that node also will crash?
>>>
>> If 1/3 zookeeper nodes crashes, the other two will take over. Kafka broker
>> will not crash. However, you should not run zookeeper and kafka on the same
>> server in production.
>>
>>> What happens if broker crashes? I suppose other two brokers will take the
>>> load.
>>>
>> Yes, the other two will take the load, but it also depends on the number
>> of partitions and how they are distributed across cluster.
>>
>> Now what happens if 2 zookeeper nodes crashes.
>>> Or if 2 brokers crashes. Will my cluster be still working in this case.
>>>
>> If 2 zookeeper nodes crash, there's no longer a majority and your cluster
>> will be down. If 2 kafka brokers crash and you have replication factor 3
>> you should be OK, but as in previous answer - it depends on the number of
>> partitions and how they're spread in the cluster.
>>
>> So basically what is the difference between zookeeper failure and
>>> server/broker failure.
>>>
>> Again, you shouldn't run zookeeper and kafka on the same server in
>> production. So the difference is that while kafka is responsible for data,
>> zookeeper is coordinating tasks between kafka nodes.
>>
>>
>>
>> On 2016.11.08 17:12, Sachin Mittal wrote:
>>
>>> Hi,
>>> We have following setup.
>>> 1. Three instances of zookeeper on three machines.
>>> 2. Three instances of kafka server on same three machines.
>>> 3. All the topics have replication factor 3.
>>>
>>> So when we create a topic on any node, i see that it gets replicated on
>>> all
>>> three instances.
>>> I also see that topic data is getting replicated to all three nodes.
>>>
>>> The data to main topic is written by three producers to which all three zk
>>> nodes config is provided in connect string.
>>>
>>> This is all working fine.
>>>
>>> The question is what happens if one of the zookeeper crashes. Will the
>>> broker on that node also will crash?
>>>
>>> What happens if broker crashes? I suppose other two brokers will take the
>>> load.
>>>
>>> Now what happens if 2 zookeeper nodes crashes.
>>> Or if 2 brokers crashes. Will my cluster be still working in this case.
>>>
>>> So basically what is the difference between zookeeper failure and
>>> server/broker failure.
>>>
>>> Thanks
>>> Sachin
>>>
>>>

Re: Understanding zookeper and kafka server failures

[Sachin Mittal <sj...@gmail.com>]

Hi,
Thanks for the reply. From one obvious reason that is server crashes then
both zookeeper and broker crashes, is there any other reason why we should
not run broker and zookeeper on same server.

If chances of server crash are extremely low can can be brought back up
quickly, then can we keep both on same server.

Thanks
Sachin



On Tue, Nov 8, 2016 at 8:59 PM, Karolis Pocius <k....@adform.com> wrote:

> The question is what happens if one of the zookeeper crashes. Will the
>> broker on that node also will crash?
>>
> If 1/3 zookeeper nodes crashes, the other two will take over. Kafka broker
> will not crash. However, you should not run zookeeper and kafka on the same
> server in production.
>
>> What happens if broker crashes? I suppose other two brokers will take the
>> load.
>>
> Yes, the other two will take the load, but it also depends on the number
> of partitions and how they are distributed across cluster.
>
> Now what happens if 2 zookeeper nodes crashes.
>> Or if 2 brokers crashes. Will my cluster be still working in this case.
>>
> If 2 zookeeper nodes crash, there's no longer a majority and your cluster
> will be down. If 2 kafka brokers crash and you have replication factor 3
> you should be OK, but as in previous answer - it depends on the number of
> partitions and how they're spread in the cluster.
>
> So basically what is the difference between zookeeper failure and
>> server/broker failure.
>>
> Again, you shouldn't run zookeeper and kafka on the same server in
> production. So the difference is that while kafka is responsible for data,
> zookeeper is coordinating tasks between kafka nodes.
>
>
>
> On 2016.11.08 17:12, Sachin Mittal wrote:
>
>> Hi,
>> We have following setup.
>> 1. Three instances of zookeeper on three machines.
>> 2. Three instances of kafka server on same three machines.
>> 3. All the topics have replication factor 3.
>>
>> So when we create a topic on any node, i see that it gets replicated on
>> all
>> three instances.
>> I also see that topic data is getting replicated to all three nodes.
>>
>> The data to main topic is written by three producers to which all three zk
>> nodes config is provided in connect string.
>>
>> This is all working fine.
>>
>> The question is what happens if one of the zookeeper crashes. Will the
>> broker on that node also will crash?
>>
>> What happens if broker crashes? I suppose other two brokers will take the
>> load.
>>
>> Now what happens if 2 zookeeper nodes crashes.
>> Or if 2 brokers crashes. Will my cluster be still working in this case.
>>
>> So basically what is the difference between zookeeper failure and
>> server/broker failure.
>>
>> Thanks
>> Sachin
>>
>>
>

Re: Understanding zookeper and kafka server failures

[Karolis Pocius <k....@adform.com>]

> The question is what happens if one of the zookeeper crashes. Will the
> broker on that node also will crash?
If 1/3 zookeeper nodes crashes, the other two will take over. Kafka 
broker will not crash. However, you should not run zookeeper and kafka 
on the same server in production.
> What happens if broker crashes? I suppose other two brokers will take the
> load.
Yes, the other two will take the load, but it also depends on the number 
of partitions and how they are distributed across cluster.

> Now what happens if 2 zookeeper nodes crashes.
> Or if 2 brokers crashes. Will my cluster be still working in this case.
If 2 zookeeper nodes crash, there's no longer a majority and your 
cluster will be down. If 2 kafka brokers crash and you have replication 
factor 3 you should be OK, but as in previous answer - it depends on the 
number of partitions and how they're spread in the cluster.

> So basically what is the difference between zookeeper failure and
> server/broker failure.
Again, you shouldn't run zookeeper and kafka on the same server in 
production. So the difference is that while kafka is responsible for 
data, zookeeper is coordinating tasks between kafka nodes.


On 2016.11.08 17:12, Sachin Mittal wrote:
> Hi,
> We have following setup.
> 1. Three instances of zookeeper on three machines.
> 2. Three instances of kafka server on same three machines.
> 3. All the topics have replication factor 3.
>
> So when we create a topic on any node, i see that it gets replicated on all
> three instances.
> I also see that topic data is getting replicated to all three nodes.
>
> The data to main topic is written by three producers to which all three zk
> nodes config is provided in connect string.
>
> This is all working fine.
>
> The question is what happens if one of the zookeeper crashes. Will the
> broker on that node also will crash?
>
> What happens if broker crashes? I suppose other two brokers will take the
> load.
>
> Now what happens if 2 zookeeper nodes crashes.
> Or if 2 brokers crashes. Will my cluster be still working in this case.
>
> So basically what is the difference between zookeeper failure and
> server/broker failure.
>
> Thanks
> Sachin
>