Re: Duplicate messages when reading with multiple consumers in same group

[Gabriel Reid <ga...@gmail.com>]

To answer my own question for people who may run into this same issue
in the future: it appears that this behavior is by design, or at least
parts of Kafka are written to work around this issue.

Looking at the class kafka.javaapi.consumer.ConsumerRebalanceListener,
it can be seen that the method beforeReleasingPartitions was added
specifically to avoid duplicate data delivery when auto-commit is
disabled in MirrorMaker. In any case, the way to work around this
issue is to implement the ConsumerRebalanceListener interface,
register it with the ConsumerConnector, and be aware that if you don't
commit the current offsets (and/or do other custom handling of state)
in the ConsumerRebalanceListener#beforeReleasingPartitions method, you
will likely receive duplicate data.

- Gabriel

On Thu, Oct 27, 2016 at 8:38 PM, Gabriel Reid <ga...@gmail.com> wrote:
> Hi,
>
> I've run into an issue using the old high-level consumer, and I'd like
> to verify that it is indeed a bug (I'm pretty sure it is), or see if
> I'm just doing something wrong.
>
> The situation is that I have two consumers (A and B) in the same
> consumer group reading from a single topic. One consumer (A) consumes
> the full content of the queue, after which the other consumer (B) is
> simply shut down (without having consumed anything). If I then attempt
> to continue reading with the first consumer (A), it begins reading
> from the start of the topic again (i.e. there is duplicate message
> delivery).
>
> What makes this particularly surprising is that it only occurs if
> auto-commit is disabled, and/or if the second consumer (B) is shut
> down after A has read the full contents of the topic.
>
> A minimal test case which demonstrates this issue is available in this
> gist: https://gist.github.com/gabrielreid/1d8a35bf3fb021bce27ceaab1b0e6152
>
> Could anyone confirm that I'm not just making a silly mistake
> somewhere which causes this behavior?
>
> Assuming that this is indeed a bug, any pointers on where to look to
> fix it would also be appreciated.
>
> Thanks,
>
> Gabriel