Reg Kafka Replication

[Balasubramanian Jayaraman <ba...@autodesk.com>]

Hi,

I have created a topic with the number of partitions as 5 and replication factor as 3. I have 5 Kafka brokers up and running.

What happens if a broker with broker id '0' is down ?

My observation is as follows.


1.       The brokers are running as before and responding to the producers as well as the consumers.

2.       A new leader is elected for all the topics/partitions for which the broker with id 0 is the leader.

3.       The ISR is updated with the available brokers (in this case 2 brokers for some topic/partition combination).

4.       The ISR has 3 brokers after the broker with id 0 is up and running again.

I have a question on this:

Why the (topics/partition) combination which has broker with id 0 in their replication list does not find a new broker and replicate the messages? Is this the intended behavior of Kafka ?

Re: Reg Kafka Replication

[Daniel Compton <de...@danielcompton.net>]

Hi Balasubramanian

> Why the (topics/partition) combination which has broker with id 0 in their replication list does not find a new broker and replicate the messages? Is this the intended behavior of Kafka ?

Do you mean, why does Broker 0 stay in the replication set for partitions when it goes down?

This seems like a design decision as it means if 0 has a transient failure, due to a network blip or a restart, it will rejoin it's partitions when it comes back up. This would result in the least network activity as it would have most of the log already on disk and would just need to catch up.

One could also imagine a scenario where a broker went down because of load. This broker's partition assignments would get reassigned to other highly loaded brokers which could cause a cascading failure (I think). 

You can reassign partitions with https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-6.ReassignPartitionsTool. It is also a very good idea to monitor the JMX bean for under replicated partitions as this would indicate the partitions at risk. 

---
Daniel

> On 30/06/2014, at 3:26 pm, Balasubramanian Jayaraman <ba...@autodesk.com> wrote:
> 
> Hi,
> 
> I have created a topic with the number of partitions as 5 and replication factor as 3. I have 5 Kafka brokers up and running.
> 
> What happens if a broker with broker id '0' is down ?
> 
> My observation is as follows.
> 
> 
> 1.       The brokers are running as before and responding to the producers as well as the consumers.
> 
> 2.       A new leader is elected for all the topics/partitions for which the broker with id 0 is the leader.
> 
> 3.       The ISR is updated with the available brokers (in this case 2 brokers for some topic/partition combination).
> 
> 4.       The ISR has 3 brokers after the broker with id 0 is up and running again.
> 
> I have a question on this:
> 
>