I found kafka lsot message

[ya...@sina.com]

Hello every body,I build a kafka cluster(include 5 domains) use kafka_2.11-0.10.0.0 and kafka-python-1.3.1.I create a topic by 100 partitions and 2 replicate, then use 20 consumers to receive message.
But, I found sometime the kafka lost message! I found some partition's offsite lost at consumer.After, I make a global index for every message from producer for confirm this problem, and I also found the global index had been break!
Why the kafka lost message?  What can I do to fix the problem?
Thanks!Jerry

--------------------------------

Re: I found kafka lsot message

[Guozhang Wang <wa...@gmail.com>]

Jerry,

Message lost scenarios usually are due to producer and consumer
mis-configured. Have you read about the client configs web docs?

http://kafka.apache.org/documentation#producerconfigs

http://kafka.apache.org/documentation#newconsumerconfigs


If not I'd suggest you reading those first and see if you can tune some of
these configs to have better delivery guarantees.

Guozhang


On Fri, Oct 7, 2016 at 9:48 PM, <ya...@sina.com> wrote:

> Hello every body,I build a kafka cluster(include 5 domains) use
> kafka_2.11-0.10.0.0 and kafka-python-1.3.1.I create a topic by 100
> partitions and 2 replicate, then use 20 consumers to receive message.
> But, I found sometime the kafka lost message! I found some partition's
> offsite lost at consumer.After, I make a global index for every message
> from producer for confirm this problem, and I also found the global index
> had been break!
> Why the kafka lost message?  What can I do to fix the problem?
> Thanks!Jerry
>
> --------------------------------
>
>


-- 
-- Guozhang