You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Michaud, Ben A" <be...@optum.com> on 2017/02/10 18:40:05 UTC

Is it possible to receive duplicate messages from Kafka when...

Is it possible to receive duplicate messages from Kafka 0.9.0.1 or 0.10.1.0 when you have a topic with three partitions, and one consumer group with three consumer clients. One client stops consuming and is taken offline. These clients do not commit offset immediately, but the offsets are committed automatically after a default wait time setting. The partition assigned to the client that goes down is moved to another client in the same group automatically.

Meanwhile, the client that went down gets some TLC, still holds some messages that were retrieved but never fully processed. When it comes back up, it happily completes processing the data and writes it to an HDFS.

Will the second client be given uncommitted messages that the first client had already received, but never committed? This would result in duplicate messages on HDFS, which is what we witnessed this week when just such a thing happened.

Regards,
Ben



This e-mail, including attachments, may include confidential and/or
proprietary information, and may be used only by the person or entity
to which it is addressed. If the reader of this e-mail is not the intended
recipient or his or her authorized agent, the reader is hereby notified
that any dissemination, distribution or copying of this e-mail is
prohibited. If you have received this e-mail in error, please notify the
sender by replying to this message and delete this e-mail immediately.

Re: Is it possible to receive duplicate messages from Kafka when...

Posted by "Matthias J. Sax" <ma...@confluent.io>.
Yes. That could happen.

Kafka provided at-least-once processing semantics if you commit messages
after processing.

You can avoid duplicates, if you commit offsets before processing, but
this might result in data loss.

Getting exactly-once is quite hard, and you will need to build your own
de-duplication logic.


-Matthias

On 2/10/17 10:40 AM, Michaud, Ben A wrote:
> Is it possible to receive duplicate messages from Kafka 0.9.0.1 or 0.10.1.0 when you have a topic with three partitions, and one consumer group with three consumer clients. One client stops consuming and is taken offline. These clients do not commit offset immediately, but the offsets are committed automatically after a default wait time setting. The partition assigned to the client that goes down is moved to another client in the same group automatically.
> 
> Meanwhile, the client that went down gets some TLC, still holds some messages that were retrieved but never fully processed. When it comes back up, it happily completes processing the data and writes it to an HDFS.
> 
> Will the second client be given uncommitted messages that the first client had already received, but never committed? This would result in duplicate messages on HDFS, which is what we witnessed this week when just such a thing happened.
> 
> Regards,
> Ben
> 
> 
> 
> This e-mail, including attachments, may include confidential and/or
> proprietary information, and may be used only by the person or entity
> to which it is addressed. If the reader of this e-mail is not the intended
> recipient or his or her authorized agent, the reader is hereby notified
> that any dissemination, distribution or copying of this e-mail is
> prohibited. If you have received this e-mail in error, please notify the
> sender by replying to this message and delete this e-mail immediately.
>