Re: Question: Data Loss and Data Duplication in Kafka

[Jayesh Thakrar <j_...@yahoo.com.INVALID>]

Thanks Radha Krishna!
So from what I understand, data loss can happen at producer due to BufferExhaustedException, failure to close/terminate producer and due to communication errors (first figure below).
And at the broker during unclean leader election (i.e. electing a leader that was not in ISR) or if a leader crashes and the min.insync.replicas configuration for the topic/broker was 1, in which case there is a potential loss of small number of messages as contained within the replica.lag.time.max.ms duration.  
Also wondering if you and the community can validate these two pictures below?









      From: R Krishna <kr...@gmail.com>
 To: users@kafka.apache.org; Jayesh Thakrar <j_...@yahoo.com> 
 Sent: Tuesday, August 30, 2016 2:02 AM
 Subject: Re: Question: Data Loss and Data Duplication in Kafka
   
Experimenting with kafka myself, and found timeouts/batch expiry (valid and invalid configurations), and max retries also can drop messages unless you handle and log them gracefully. There are also a bunch of org.apache.kafka.common.KafkaException hierarchy exceptions some of which are thrown for valid reasons but also drop messages like size of messages, buffer size, etc.,. 

On Sun, Aug 28, 2016 at 1:55 AM, Jayesh Thakrar <j_...@yahoo.com.invalid> wrote:

I am looking at ways how one might have data loss and duplication in a Kafka cluster and need some help/pointers/discussions.
So far, here's what I have come up with:
Loss at producer-sideSince the data send call is actually adding data to a cache/buffer, a crash of the producer can potentially result in data loss.Another scenario for data loss is a producer exiting without closing the producer connection.
Loss at broker-sideI think there are several situations here - all of which are triggered by a broker or controller crash or network issues with zookeepers (kind of simulating broker crashes). 
If I understand correctly, KAFKA-1211 (https://issues.apache.org/ jira/browse/KAFKA-1211) implies that when acks is set to 0/1 and the leader crashes, there is a probability of data loss. Hopefully implementation of leader generation will help avoid this (https://issues.apache.org/ jira/browse/KAFKA-1211? focusedCommentId=15402622& page=com.atlassian.jira. plugin.system.issuetabpanels: comment-tabpanel#comment- 15402622)
And a unique situation as described in KAFKA-3410 (https://issues.apache.org/ jira/browse/KAFKA-3410) can cause broker or cluster shutdown leading to data loss as described in KAFKA-3924 (resolved in 0.10.0.1).
And data duplication can attributed primarily to consumer offset management which is done at batch/periodic intervals.
Can anyone think or know of any other scenarios?
Thanks,Jayesh







-- 
Radha Krishna, Proddaturi253-234-5657