You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Fathima Amara <fa...@wso2.com> on 2017/05/16 09:09:31 UTC

Data loss after a Kafka Broker restart scenario

Hi all,

I am using Kafka 2.11-0.10.0.1 and Zookeeper 3.4.8.
I have a cluster of 4 servers(A,B,C,D) running one kafka broker on each of them and, one zookeeper server on server A. Data is initially produced from server A using a Kafka Producer and it goes through servers B,C,D being subjected to processing and finally reaches server A again(gets consumed using a Kafka Consumer). 

Topics created on the end of each process has 2 partitions with a replication-factor of 3. Other configurations include,
unclean.leader.election.enable=false
acks=all
retries=0
I let the producer run for a while in server A, then kill one of the Kafka brokers on the cluster(B,C,D) while data processing takes place and restart it. When consuming from the end of server A, I notice a considerable amount of data lost which varies on each run! ex:- on an input of 1 million events 5930 events are lost.

Is the reason for this the Kafka Producer not guaranteeing Exactly-once processing or is this due to some other reason? what other reasons cause data loss?

Re: Data loss after a Kafka Broker restart scenario

Posted by Jun Rao <ju...@confluent.io>.
Hi, Fathima,

Yes, the most efficient way to verify if a message is sent successfully is
through the producer callback. You can take a look at PrintInfoCallback in
org.apache.kafka.toolsVerifiableProducer as an example. Our system tests
use that to verify if any data loss has occurred.

Thanks,

Jun

On Mon, May 22, 2017 at 2:48 AM, Fathima Amara <fa...@wso2.com> wrote:

> Hi Jun,
>
> Do you mean by using CallBack mechanism? Since I am new to kafka would you
> mind directing me how to do it if it's not to be done using CallBack?
>
> Fathima.
>

Re: Data loss after a Kafka Broker restart scenario

Posted by Fathima Amara <fa...@wso2.com>.
Hi Jun,

Do you mean by using CallBack mechanism? Since I am new to kafka would you mind directing me how to do it if it's not to be done using CallBack? 

Fathima.

Re: Data loss after a Kafka Broker restart scenario

Posted by Jun Rao <ju...@confluent.io>.
Hi, Fathima,

Did you check if produced messages are acked successfully? Only
successfully acked messages are guaranteed to be preserved on the broker.

Thanks,

Jun

On Thu, May 18, 2017 at 12:10 AM, Fathima Amara <fa...@wso2.com> wrote:

> Hi Jun,
>
> Thanks alot for the reply. As suggested, I tried running my application in
> the trunk. But I still do encounter data loss! Are there any specific
> configs that I need to change from its default value? What could be the
> reason for this?
>
> Fathima
>

Re: Data loss after a Kafka Broker restart scenario

Posted by Fathima Amara <fa...@wso2.com>.
Hi Jun,

Thanks alot for the reply. As suggested, I tried running my application in the trunk. But I still do encounter data loss! Are there any specific configs that I need to change from its default value? What could be the reason for this?

Fathima

Re: Data loss after a Kafka Broker restart scenario

Posted by Jun Rao <ju...@confluent.io>.
Hi, Fathima,

There is a known data loss issue that's described in KIP-101 (
https://cwiki.apache.org/confluence/display/KAFKA/KIP-101+-+Alter+Replication+Protocol+to+use+Leader+Epoch+rather+than+High+Watermark+for+Truncation).
The issue happens rarely, but has been exposed in some of our system tests
that mimic the testing that you have been doing. KIP-101 has been fixed in
trunk and will be included in the next release (0.11.0.0) in June. Since
then, we haven't observed similar data loss issue in our system tests.

If you want to test this out now, perhaps you could rerun your test in
trunk.

Thanks,

Jun

On Tue, May 16, 2017 at 2:09 AM, Fathima Amara <fa...@wso2.com> wrote:

>
> Hi all,
>
> I am using Kafka 2.11-0.10.0.1 and Zookeeper 3.4.8.
> I have a cluster of 4 servers(A,B,C,D) running one kafka broker on each of
> them and, one zookeeper server on server A. Data is initially produced from
> server A using a Kafka Producer and it goes through servers B,C,D being
> subjected to processing and finally reaches server A again(gets consumed
> using a Kafka Consumer).
>
> Topics created on the end of each process has 2 partitions with a
> replication-factor of 3. Other configurations include,
> unclean.leader.election.enable=false
> acks=all
> retries=0
> I let the producer run for a while in server A, then kill one of the Kafka
> brokers on the cluster(B,C,D) while data processing takes place and restart
> it. When consuming from the end of server A, I notice a considerable amount
> of data lost which varies on each run! ex:- on an input of 1 million events
> 5930 events are lost.
>
> Is the reason for this the Kafka Producer not guaranteeing Exactly-once
> processing or is this due to some other reason? what other reasons cause
> data loss?
>