Re: Painfully slow kafka recovery / cluster breaking

[Jörg Wagner <jo...@1und1.de>]

Just a little feedback on our issue(s) as FYI to whoever is interested.

It basically all boiled down to the configuration of topics. We noticed 
while performance testing (or trying to ;) ) that the partitioning was 
most critical to us.

We originally followed the linkedin recommendation and used 600 
partitions for our main topic. Testing that, the replicas always went 
out of sync within a short timeframe, leaders could not be determined 
and the cluster failed horribly (even writing several hundred lines of 
logs within a 1/100th second).

So for our 27 log.dirs (= disks) we went with 27 partitions. And voilá: 
we could use kafka with around 35k requests per second (via an 
application accessing it). Kafka stayed stable.

Currently we are testing with 81 partitions (27*3) and it's running 
well. No issues so far, replicas in sync and up to 50k requests per second.

Cheers

On 25.08.2015 15:18, Jörg Wagner wrote:
> So okay, this is a little embarassing but the core of the issue was 
> that max open files was not set correctly for kafka. It was not an 
> oversight, but a few things together caused that the system 
> configuration was not changed correctly, resulting in the default value.
>
> No wonder that kafka behaved strangely everytime we had enough data in 
> log.dirs and connections.
>
> Anyhow, that doesn't seem to be the last problem. The brokers get in 
> sync with each other (within an expected time frame), everything seems 
> fine.
>
> After a little stress testing, the kafka cluster falls apart (around 
> 40k requests/s). Using topics describe we can see leaders missing 
> (e.g. from 1,2,3 only 1 and 3 are leading partitions, although 
> zookeeper lists all under /brokers/ids). This ultimately results in 
> partitions being unavailable and massive "leader not local" spam in 
> the logs.
>
> What are we missing?
>
> Cheers
> Jörg
>
> On 24.08.2015 10:31, Jörg Wagner wrote:
>> Thank you for your answers.
>>
>> @Raja
>> No, it also seems to happen if we stop kafka completely clean.
>>
>> @Gwen
>> I was testing the situation with num.replica.fetchers set higher. If 
>> you say that was the right direction, I will try it again. What would 
>> be a good setting? I went with 50 which seemed reasonable (having 27 
>> single disks).
>> How long should it take to get complete ISR?
>>
>> Regarding no Data flowing into kafka: I just wanted to point out that 
>> the setup is not yet live. So we can completely stop the usage of 
>> kafka, and it should possibly get into sync faster without a steady 
>> stream of new messages.
>> Kafka itself is working fine during this on the other hand, "just" 
>> missing ISR, hence redundancy. If I stop another broker (clean!) 
>> though, it tends to happen that the expected number of partitions 
>> have Leader -1; which should not happen as I assume.
>>
>> Cheers
>> Jörg
>>
>> On 21.08.2015 19:18, Rajasekar Elango wrote:
>>> We are seeing same behavior in 5 broker cluster when losing one broker.
>>>
>>> In our case, we are losing broker as well as kafka data dir.
>>>
>>> Jörg Wagner,
>>>
>>> Are you losing just broker or kafka data dir as well?
>>>
>>> Gwen,
>>>
>>> We have also observed that latency of messages arriving at consumers 
>>> goes
>>> up by 10x when we lose a broker. Is it because the broker is busy with
>>> handling failed fetch requests and loaded with more data thats 
>>> slowing down
>>> the writes ? Also, if we had simply lost the broker not the data dir,
>>> impact would have been minimal?
>>>
>>> Thanks,
>>> Raja.
>>>
>>>
>>>
>>> On Fri, Aug 21, 2015 at 12:31 PM, Gwen Shapira <gw...@confluent.io> 
>>> wrote:
>>>
>>>> By default, num.replica.fetchers = 1. This means only one thread 
>>>> per broker
>>>> is fetching data from leaders. This means it make take a while for the
>>>> recovering machine to catch up and rejoin the ISR.
>>>>
>>>> If you have bandwidth to spare, try increasing this value.
>>>>
>>>> Regarding "no data flowing into kafka" - If you have 3 replicas and 
>>>> only
>>>> one is down, I'd expect writes to continue to the new leader even 
>>>> if one
>>>> replica is not in the ISR yet. Can you see that a new leader is 
>>>> elected?
>>>>
>>>> Gwen
>>>>
>>>> On Fri, Aug 21, 2015 at 6:50 AM, Jörg Wagner <jo...@1und1.de>
>>>> wrote:
>>>>
>>>>> Hey everyone,
>>>>>
>>>>> here's my crosspost from irc.
>>>>>
>>>>> Our setup:
>>>>> 3 kafka 0.8.2 brokers with zookeeper, powerful hardware (20 cores, 27
>>>>> logdisks each). We use a handful of topics, but only one topic is
>>>> utilized
>>>>> heavily. It features a replication of 2 and 600 partitions.
>>>>>
>>>>> Our issue:
>>>>> If one kafka was down, it takes very long ( from 1 to >10 hours) 
>>>>> to show
>>>>> that all partitions have all isr again. This seems to heavily 
>>>>> depend on
>>>> the
>>>>> amount of data which is in the log.dirs (I have configured 27 
>>>>> threads -
>>>> one
>>>>> for each dir featuring a own drive).
>>>>> This all takes this long while there is NO data flowing into kafka.
>>>>>
>>>>> We seem to be missing something critical here. It might be some 
>>>>> option
>>>> set
>>>>> wrong, or are we thinking wrong and it's not critical to have the
>>>> replicas
>>>>> in sync.
>>>>>
>>>>> Any pointers would be great.
>>>>>
>>>>> Cheers
>>>>> Jörg
>>>>>
>>>
>>>
>>
>

-- 
Mit freundlichem Gruß

Jörg Wagner

  
Mobile & Services

1&1 Internet AG | Sapporobogen 6-8 | 80637 München | Germany
Phone: +49 89 14339 324
E-Mail: joerg.wagner1@1und1.de | Web: www.1und1.de

Hauptsitz Montabaur, Amtsgericht Montabaur, HRB 6484

Vorstand: Ralph Dommermuth, Frank Einhellinger, Robert Hoffmann, Andreas Hofmann, Markus Huhn, Hans-Henning Kettler, Uwe Lamnek, Jan Oetjen, Christian Würst
Aufsichtsratsvorsitzender: Michael Scheeren

Member of United Internet

Re: Painfully slow kafka recovery / cluster breaking

[Rajasekar Elango <re...@salesforce.com>]

Thanks for updates Jörg. It's very useful.

Thanks,
Raja.

On Wed, Aug 26, 2015 at 8:58 AM, Jörg Wagner <jo...@1und1.de> wrote:

> Just a little feedback on our issue(s) as FYI to whoever is interested.
>
> It basically all boiled down to the configuration of topics. We noticed
> while performance testing (or trying to ;) ) that the partitioning was most
> critical to us.
>
> We originally followed the linkedin recommendation and used 600 partitions
> for our main topic. Testing that, the replicas always went out of sync
> within a short timeframe, leaders could not be determined and the cluster
> failed horribly (even writing several hundred lines of logs within a
> 1/100th second).
>
> So for our 27 log.dirs (= disks) we went with 27 partitions. And voilá: we
> could use kafka with around 35k requests per second (via an application
> accessing it). Kafka stayed stable.
>
> Currently we are testing with 81 partitions (27*3) and it's running well.
> No issues so far, replicas in sync and up to 50k requests per second.
>
> Cheers
>
> On 25.08.2015 15:18, Jörg Wagner wrote:
>
>> So okay, this is a little embarassing but the core of the issue was that
>> max open files was not set correctly for kafka. It was not an oversight,
>> but a few things together caused that the system configuration was not
>> changed correctly, resulting in the default value.
>>
>> No wonder that kafka behaved strangely everytime we had enough data in
>> log.dirs and connections.
>>
>> Anyhow, that doesn't seem to be the last problem. The brokers get in sync
>> with each other (within an expected time frame), everything seems fine.
>>
>> After a little stress testing, the kafka cluster falls apart (around 40k
>> requests/s). Using topics describe we can see leaders missing (e.g. from
>> 1,2,3 only 1 and 3 are leading partitions, although zookeeper lists all
>> under /brokers/ids). This ultimately results in partitions being
>> unavailable and massive "leader not local" spam in the logs.
>>
>> What are we missing?
>>
>> Cheers
>> Jörg
>>
>> On 24.08.2015 10:31, Jörg Wagner wrote:
>>
>>> Thank you for your answers.
>>>
>>> @Raja
>>> No, it also seems to happen if we stop kafka completely clean.
>>>
>>> @Gwen
>>> I was testing the situation with num.replica.fetchers set higher. If you
>>> say that was the right direction, I will try it again. What would be a good
>>> setting? I went with 50 which seemed reasonable (having 27 single disks).
>>> How long should it take to get complete ISR?
>>>
>>> Regarding no Data flowing into kafka: I just wanted to point out that
>>> the setup is not yet live. So we can completely stop the usage of kafka,
>>> and it should possibly get into sync faster without a steady stream of new
>>> messages.
>>> Kafka itself is working fine during this on the other hand, "just"
>>> missing ISR, hence redundancy. If I stop another broker (clean!) though, it
>>> tends to happen that the expected number of partitions have Leader -1;
>>> which should not happen as I assume.
>>>
>>> Cheers
>>> Jörg
>>>
>>> On 21.08.2015 19:18, Rajasekar Elango wrote:
>>>
>>>> We are seeing same behavior in 5 broker cluster when losing one broker.
>>>>
>>>> In our case, we are losing broker as well as kafka data dir.
>>>>
>>>> Jörg Wagner,
>>>>
>>>> Are you losing just broker or kafka data dir as well?
>>>>
>>>> Gwen,
>>>>
>>>> We have also observed that latency of messages arriving at consumers
>>>> goes
>>>> up by 10x when we lose a broker. Is it because the broker is busy with
>>>> handling failed fetch requests and loaded with more data thats slowing
>>>> down
>>>> the writes ? Also, if we had simply lost the broker not the data dir,
>>>> impact would have been minimal?
>>>>
>>>> Thanks,
>>>> Raja.
>>>>
>>>>
>>>>
>>>> On Fri, Aug 21, 2015 at 12:31 PM, Gwen Shapira <gw...@confluent.io>
>>>> wrote:
>>>>
>>>> By default, num.replica.fetchers = 1. This means only one thread per
>>>>> broker
>>>>> is fetching data from leaders. This means it make take a while for the
>>>>> recovering machine to catch up and rejoin the ISR.
>>>>>
>>>>> If you have bandwidth to spare, try increasing this value.
>>>>>
>>>>> Regarding "no data flowing into kafka" - If you have 3 replicas and
>>>>> only
>>>>> one is down, I'd expect writes to continue to the new leader even if
>>>>> one
>>>>> replica is not in the ISR yet. Can you see that a new leader is
>>>>> elected?
>>>>>
>>>>> Gwen
>>>>>
>>>>> On Fri, Aug 21, 2015 at 6:50 AM, Jörg Wagner <jo...@1und1.de>
>>>>> wrote:
>>>>>
>>>>> Hey everyone,
>>>>>>
>>>>>> here's my crosspost from irc.
>>>>>>
>>>>>> Our setup:
>>>>>> 3 kafka 0.8.2 brokers with zookeeper, powerful hardware (20 cores, 27
>>>>>> logdisks each). We use a handful of topics, but only one topic is
>>>>>>
>>>>> utilized
>>>>>
>>>>>> heavily. It features a replication of 2 and 600 partitions.
>>>>>>
>>>>>> Our issue:
>>>>>> If one kafka was down, it takes very long ( from 1 to >10 hours) to
>>>>>> show
>>>>>> that all partitions have all isr again. This seems to heavily depend
>>>>>> on
>>>>>>
>>>>> the
>>>>>
>>>>>> amount of data which is in the log.dirs (I have configured 27 threads
>>>>>> -
>>>>>>
>>>>> one
>>>>>
>>>>>> for each dir featuring a own drive).
>>>>>> This all takes this long while there is NO data flowing into kafka.
>>>>>>
>>>>>> We seem to be missing something critical here. It might be some option
>>>>>>
>>>>> set
>>>>>
>>>>>> wrong, or are we thinking wrong and it's not critical to have the
>>>>>>
>>>>> replicas
>>>>>
>>>>>> in sync.
>>>>>>
>>>>>> Any pointers would be great.
>>>>>>
>>>>>> Cheers
>>>>>> Jörg
>>>>>>
>>>>>>
>>>>
>>>>
>>>
>>
> --
> Mit freundlichem Gruß
>
> Jörg Wagner
>
>  Mobile & Services
>
> 1&1 Internet AG | Sapporobogen 6-8 | 80637 München | Germany
> Phone: +49 89 14339 324
> E-Mail: joerg.wagner1@1und1.de | Web: www.1und1.de
>
> Hauptsitz Montabaur, Amtsgericht Montabaur, HRB 6484
>
> Vorstand: Ralph Dommermuth, Frank Einhellinger, Robert Hoffmann, Andreas
> Hofmann, Markus Huhn, Hans-Henning Kettler, Uwe Lamnek, Jan Oetjen,
> Christian Würst
> Aufsichtsratsvorsitzender: Michael Scheeren
>
> Member of United Internet
>
>


-- 
Thanks,
Raja.