Consumer reading from same topic twice without commiting

[dinesh kumar <di...@gmail.com>]

Hi,
I am trying to understand the behavior of Kafka Consumer in Java.

Consider a scenario where a consumer is reading from a partition with
auto.comiit disabled and only one partition in the topic. The consumer
reads a message 'A' and sleep for some time, say 10 seconds. Then reads
from the same parition once again from the without commiting the offsets of
 the previous read a message 'B'.

My assumption was since there was no commit of the previous offset we will
get the same message message 'A' and message 'B' should be the same.

I ran a simple code with this assumption but I was suprised to see message
'A' and message 'B' were different (Producer produced Message 'B' after
message 'A').

Can someone explain this behavior? Is there a way to make the consumer
return message 'A' instead of message 'B' without restarting the consumer
(which will trigger a repartition in a multi partition/ multi consumer
scenario), say some parameter or a method call to rewind locally?


Thanks,
Dinesh

Re: Consumer reading from same topic twice without commiting

[Chia-Chun Shih <ch...@gmail.com>]

Auto Commit is to sync offset to zookeeper periodically. Without auto
commit, consumer still maintains an offset locally. It explains this
behavior.

To be able to rewind, you need to use SimpleConsumer.
2014/11/15 上午2:18 於 "dinesh kumar" <di...@gmail.com> 寫道：

> Hi,
> I am trying to understand the behavior of Kafka Consumer in Java.
>
> Consider a scenario where a consumer is reading from a partition with
> auto.comiit disabled and only one partition in the topic. The consumer
> reads a message 'A' and sleep for some time, say 10 seconds. Then reads
> from the same parition once again from the without commiting the offsets of
>  the previous read a message 'B'.
>
> My assumption was since there was no commit of the previous offset we will
> get the same message message 'A' and message 'B' should be the same.
>
> I ran a simple code with this assumption but I was suprised to see message
> 'A' and message 'B' were different (Producer produced Message 'B' after
> message 'A').
>
> Can someone explain this behavior? Is there a way to make the consumer
> return message 'A' instead of message 'B' without restarting the consumer
> (which will trigger a repartition in a multi partition/ multi consumer
> scenario), say some parameter or a method call to rewind locally?
>
>
> Thanks,
> Dinesh
>