Kafka High Level Consumer Message Loss?

[Allen Wang <aw...@netflix.com.INVALID>]

We have two applications that consume all messages from one Kafka cluster.
We found that the MessagesPerSec metric started to diverge after some time.
One of them matches the MessagesInPerSec metric from the Kafka broker,
while the other is lower than the broker metric and appears to have some
message loss.

Both of them have the same OwnedPartitionsCount.

Both of them have 0 MaxLag.

How is that possible? Anything we should look at? Is the MaxLag metric not
telling the truth?

Thanks,
Allen

Re: Kafka High Level Consumer Message Loss?

[Mayuresh Gharat <gh...@gmail.com>]

Can you confirm that you are not actually seeing the messages on the
lagging broker ?
Because if the Max Lag is 0 it should mean that consumer has read the
offsets till log end offset of the broker.

Thanks,

Mayuresh

On Fri, Jul 10, 2015 at 8:29 PM, Allen Wang <aw...@netflix.com.invalid>
wrote:

> We have two applications that consume all messages from one Kafka cluster.
> We found that the MessagesPerSec metric started to diverge after some time.
> One of them matches the MessagesInPerSec metric from the Kafka broker,
> while the other is lower than the broker metric and appears to have some
> message loss.
>
> Both of them have the same OwnedPartitionsCount.
>
> Both of them have 0 MaxLag.
>
> How is that possible? Anything we should look at? Is the MaxLag metric not
> telling the truth?
>
> Thanks,
> Allen
>



-- 
-Regards,
Mayuresh R. Gharat
(862) 250-7125