Kafka under replicated partitions..

[nitin sharma <ku...@gmail.com>]

Hi All,

I would like to know the factors that can cause followers to fall behind
and under-replicated replica getting created?

In my production system , last Friday one of the partitions fell behind and
before i could check my operation team had restarted the follower broker...
Restart did fixed the issue but i would like to know

a. the factors that can cause under replicated partitions
b. how to fix that issue? is restart an option?
c. apart from M-Bean any other way to get to know under-replicated
partition.?


Regards,
Nitin Kumar Sharma.

Re: Kafka under replicated partitions..

[Guozhang Wang <wa...@gmail.com>]

Hi Nitin,

a. The follower replica will be kicked out of the ISR (i.e. causing the
partition to be under-replicated) when 1) it has lagged much behind the
leader replica in terms of number of messages (controlled by config
replica.lag.max.messages), or 2) it has not fetched from leader for some
long time (controlled by config replica.socket.timeout.ms). Case 1) can
usually be triggered when the producer's throughput to the leader replica
is too high for the follower to keep up, or if one batch of producer's
messages contains too many messages than the lag threshold. Case 2) can be
triggered when the follower is faulty (long GC, etc).

You can find the definitions of these configs here:

http://kafka.apache.org/documentation.html#brokerconfigs

b. For case 2) you usually need to restart the faulty follower, for case 1)
once the producer throughput dropped to normal it should be able to catch
up later. Also to prevent it happening you can tune these two configs a bit.

c. You can use the topic command to list all under-replicated partitions in
real time:

http://kafka.apache.org/documentation.html#basic_ops_add_topic (use
bin/kafka-topics.sh --list)


Guozhang


On Mon, Jan 26, 2015 at 12:40 PM, nitin sharma <ku...@gmail.com>
wrote:

> Hi All,
>
> I would like to know the factors that can cause followers to fall behind
> and under-replicated replica getting created?
>
> In my production system , last Friday one of the partitions fell behind and
> before i could check my operation team had restarted the follower broker...
> Restart did fixed the issue but i would like to know
>
> a. the factors that can cause under replicated partitions
> b. how to fix that issue? is restart an option?
> c. apart from M-Bean any other way to get to know under-replicated
> partition.?
>
>
> Regards,
> Nitin Kumar Sharma.
>



-- 
-- Guozhang