How Kafka Manages Network Partition

[Isuru Boyagane <is...@cse.mrt.ac.lk>]

Can anyone clarify how Kafka manages below network partition?

Say we have this configuration before the network partition,

   - Kafka cluster has three brokers (say broker_0, broker_1, broker2).
   - broker_1 is the leader.
   - ISR has been reduced to broker_1 and broker_2.
   - There is a Zookeeper cluster.

Now if a network partition happens such that,

   1. broker_1(leader), broker_2 in a separate partition
   2. broker_0 and Zookeeper in another partition

(An image clarifying the network partition scenario is attached herewith.)

Will there be two leaders?
If so, How they continue when the partition is resolved?

Thanks and Regards.

Re: How Kafka Manages Network Partition

[Isuru Boyagane <is...@cse.mrt.ac.lk>]

Ok. If min.insync.replicas is 1 and the acknowledgment method is acks=1,


   - Zookeeper thinks that broker_1 and broker_2 are dead and elect
   broker_0 as the new leader.
   - But, broker_2 can still send fetch requests to broker_1 and original
   ISR (broker_1, broker_2) will remain the same.
   - In that case, until every client starts writing to broker_0, both
   broker_1 and broker_0 will act as leaders.


Isn't this what happens?

On Mon, 23 Sep 2019 at 17:34, Karolis Pocius
<ka...@sentiance.com.invalid> wrote:

> AFAIK there won't be two leaders. Once brokers lose connection with
> ZooKeeper, a new leader will be elected (whichever can still access
> ZooKeeper) and the remaining brokers will fall behind in replication.
>
> Now depending on your config, there might be several other issues: not
> enough replicas to satisfy ISR requirement, so no new writes; if the broker
> that becomes a new leader was already behind in replication and unclean
> leader election is enabled, you might have some data loss, etc.
>
> On Mon, Sep 23, 2019 at 2:52 PM Isuru Boyagane <
> isuruboyagane.16@cse.mrt.ac.lk> wrote:
>
> > Can anyone clarify how Kafka manages below network partition?
> >
> > Say we have this configuration before the network partition,
> >
> >    - Kafka cluster has three brokers (say broker_0, broker_1, broker2).
> >    - broker_1 is the leader.
> >    - ISR has been reduced to broker_1 and broker_2.
> >    - There is a Zookeeper cluster.
> >
> > Now if a network partition happens such that,
> >
> >    1. broker_1(leader), broker_2 in a separate partition
> >    2. broker_0 and Zookeeper in another partition
> >
> > (An image clarifying the network partition scenario is attached
> herewith.)
> >
> > Will there be two leaders?
> > If so, How they continue when the partition is resolved?
> >
> > Thanks and Regards.
> >
> >
>


-- 
Best Regards

Isuru Boyagane
Undergraduate
Department of Computer Science & Engineering
University of Moratuwa

Re: How Kafka Manages Network Partition

[Karolis Pocius <ka...@sentiance.com.INVALID>]

AFAIK there won't be two leaders. Once brokers lose connection with
ZooKeeper, a new leader will be elected (whichever can still access
ZooKeeper) and the remaining brokers will fall behind in replication.

Now depending on your config, there might be several other issues: not
enough replicas to satisfy ISR requirement, so no new writes; if the broker
that becomes a new leader was already behind in replication and unclean
leader election is enabled, you might have some data loss, etc.

On Mon, Sep 23, 2019 at 2:52 PM Isuru Boyagane <
isuruboyagane.16@cse.mrt.ac.lk> wrote:

> Can anyone clarify how Kafka manages below network partition?
>
> Say we have this configuration before the network partition,
>
>    - Kafka cluster has three brokers (say broker_0, broker_1, broker2).
>    - broker_1 is the leader.
>    - ISR has been reduced to broker_1 and broker_2.
>    - There is a Zookeeper cluster.
>
> Now if a network partition happens such that,
>
>    1. broker_1(leader), broker_2 in a separate partition
>    2. broker_0 and Zookeeper in another partition
>
> (An image clarifying the network partition scenario is attached herewith.)
>
> Will there be two leaders?
> If so, How they continue when the partition is resolved?
>
> Thanks and Regards.
>
>