You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by 许志峰 <zh...@gmail.com> on 2018/03/01 08:59:26 UTC
When a broker down, Producer LOST messages!
Hi all,
I have a kafka cluster with 3 nodes: node1, node2, node3
*kafka version is 0.8.2.1, which I can not change!*
A Producer writes msg to kafka, its code framework is like this in
pseudo-code:
Properties props = new Properties();props.put("bootstrap.servers",
"node1:9092,node2:9092,node3:9092");props.put(key serializer and value
serializer);
for(i = 0; i < 10; ++i){
producer = new Producer(props);
msg = "this is msg " + i;
producer.send(msg);
producer.close()
}
After the first 4 messages are send successfully, I killed broker on
node1. the 5th and 6th messages are LOST.
Producer first get the broker list from the PropertyConfig, i.e. [node1,
node2, node3], then* producer choose one broker, connect with it and get
METADATA from it. *
* I heard that when one broker in the list is unavailable, the kafka client
will change to another *
*But in my case, **if the broker choose node1, which is already dead, it
will get a Fetch MetaData Timeout Exception and STOPPED! msg is not writed
into Kafka. *
*Attached is the complete Log. you can only focus on the colorful lines.*
you can see that, I wrote 10 msgs to Kafka, the first 4 succeed, when I
kill one broker, msg5 and msg6 are LOST, because the choose NODE1,
msg7,8,9,10 are succeed because they did not choose node1.
I checkout the Kafka source codes and get nothing.
Do anybody know the reason? where are the related classes/functions
located in the source code?
Any clue will be appreciated!
Re: When a broker down, Producer LOST messages!
Posted by 许志峰 <zh...@gmail.com>.
Hi,
Thank you for you reply.
It looks like the attachment is fine. I have a replication-factor of 3.
The producer can not even get the METADATA from a broker(when it chooses a
dead broker), let alone sending messages and wait for all brokers to ACK.
Best regards!
2018-03-01 17:21 GMT+08:00 Thomas Aley <Th...@ibm.com>:
> Hi,
>
> The log is not attached. I'm assuming your topic has a replication factor
> greater than 1 so that it is available from another Broker if the
> partition leader fails. Try adding
>
> props.put("acks", "all");
>
> to your producer and run your experiment again. If you configured your
> topic to have --replication-factor 3 and your brokers (or topic itself) is
> configured with min.insync.replicas=2 for example then your producer will
> require acknowledgment of receipt of each message from 2 of your 3 brokers
> with the 'acks=all' property in use making your topic resilient.
>
> Hope this helps,
>
> Tom Aley
> thomas.aley@ibm.com
>
>
>
> From: "许志峰" <zh...@gmail.com>
> To: users@kafka.apache.org
> Date: 01/03/2018 08:59
> Subject: When a broker down, Producer LOST messages!
>
>
>
> Hi all,
>
>
> I have a kafka cluster with 3 nodes: node1, node2, node3
>
> kafka version is 0.8.2.1, which I can not change!
>
> A Producer writes msg to kafka, its code framework is like this in
> pseudo-code:
> Properties props = new Properties();
> props.put("bootstrap.servers", "node1:9092,node2:9092,node3:9092");
> props.put(key serializer and value serializer);
>
> for(i = 0; i < 10; ++i){
> producer = new Producer(props);
> msg = "this is msg " + i;
> producer.send(msg);
> producer.close()
> }
>
> After the first 4 messages are send successfully, I killed broker on
> node1. the 5th and 6th messages are LOST.
>
> Producer first get the broker list from the PropertyConfig, i.e. [node1,
> node2, node3], then producer choose one broker, connect with it and get
> METADATA from it.
>
> I heard that when one broker in the list is unavailable, the kafka client
> will change to another
>
> But in my case, if the broker choose node1, which is already dead, it will
> get a Fetch MetaData Timeout Exception and STOPPED! msg is not writed into
> Kafka.
>
>
> Attached is the complete Log. you can only focus on the colorful lines.
>
> you can see that, I wrote 10 msgs to Kafka, the first 4 succeed, when I
> kill one broker, msg5 and msg6 are LOST, because the choose NODE1,
> msg7,8,9,10 are succeed because they did not choose node1.
>
> I checkout the Kafka source codes and get nothing.
>
> Do anybody know the reason? where are the related classes/functions
> located in the source code?
>
> Any clue will be appreciated!
>
>
>
>
>
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number
> 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>
>
Re: When a broker down, Producer LOST messages!
Posted by Thomas Aley <Th...@ibm.com>.
Hi,
The log is not attached. I'm assuming your topic has a replication factor
greater than 1 so that it is available from another Broker if the
partition leader fails. Try adding
props.put("acks", "all");
to your producer and run your experiment again. If you configured your
topic to have --replication-factor 3 and your brokers (or topic itself) is
configured with min.insync.replicas=2 for example then your producer will
require acknowledgment of receipt of each message from 2 of your 3 brokers
with the 'acks=all' property in use making your topic resilient.
Hope this helps,
Tom Aley
thomas.aley@ibm.com
From: "许志峰" <zh...@gmail.com>
To: users@kafka.apache.org
Date: 01/03/2018 08:59
Subject: When a broker down, Producer LOST messages!
Hi all,
I have a kafka cluster with 3 nodes: node1, node2, node3
kafka version is 0.8.2.1, which I can not change!
A Producer writes msg to kafka, its code framework is like this in
pseudo-code:
Properties props = new Properties();
props.put("bootstrap.servers", "node1:9092,node2:9092,node3:9092");
props.put(key serializer and value serializer);
for(i = 0; i < 10; ++i){
producer = new Producer(props);
msg = "this is msg " + i;
producer.send(msg);
producer.close()
}
After the first 4 messages are send successfully, I killed broker on
node1. the 5th and 6th messages are LOST.
Producer first get the broker list from the PropertyConfig, i.e. [node1,
node2, node3], then producer choose one broker, connect with it and get
METADATA from it.
I heard that when one broker in the list is unavailable, the kafka client
will change to another
But in my case, if the broker choose node1, which is already dead, it will
get a Fetch MetaData Timeout Exception and STOPPED! msg is not writed into
Kafka.
Attached is the complete Log. you can only focus on the colorful lines.
you can see that, I wrote 10 msgs to Kafka, the first 4 succeed, when I
kill one broker, msg5 and msg6 are LOST, because the choose NODE1,
msg7,8,9,10 are succeed because they did not choose node1.
I checkout the Kafka source codes and get nothing.
Do anybody know the reason? where are the related classes/functions
located in the source code?
Any clue will be appreciated!
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU