Kafka Producer Buffer and Broker Connection Failure

[Mark Anderson <ma...@gmail.com>]

Hi,

I'm currently testing Kafka Producers in cases of broker connection failure
due to the broker process dieing or network connection timeout. I'd like to
make sure that I understand how the Producer buffer functions in this case.
Note that I have retries set to 0.

From what I can see when send is called on the Producer the Node to which
the record will be sent is the Node which is the leader for that topic
partition at the time send was called.

Therefore, if the leader for a topic partition changes while the record is
still within the Producer buffer (waiting to be sent) the record would
still be sent to the original leader from record creation time. And in this
case the request.timeout.ms would apply to the send of that record failing.

Only records created when send is called on the Producer after broker
connection failure and re-fetch of the metadata will be sent to the new
leader.

Could you please confirm my understanding is correct?

Thanks,
Mark

Re: Kafka Producer Buffer and Broker Connection Failure

[Mayuresh Gharat <gh...@gmail.com>]

Hi Mark,

The initial understanding is correct.
To understand the timeout scenarios, you might want to take a look at
KIP-91 :
https://cwiki.apache.org/confluence/display/KAFKA/KIP-91+Provide+Intuitive+User+Timeouts+in+The+Producer

If the producer sends a request to the old leader of a topic partition, it
will get a NotLeaderForPartitionException due to which it will update its
metadata and then if you have retries configured will try to send the data
to the new leader.

Thanks,

Mayuresh

On Fri, Nov 23, 2018 at 8:19 AM Mark Anderson <ma...@gmail.com> wrote:

> Hi,
>
> I'm currently testing Kafka Producers in cases of broker connection failure
> due to the broker process dieing or network connection timeout. I'd like to
> make sure that I understand how the Producer buffer functions in this case.
> Note that I have retries set to 0.
>
> From what I can see when send is called on the Producer the Node to which
> the record will be sent is the Node which is the leader for that topic
> partition at the time send was called.
>
> Therefore, if the leader for a topic partition changes while the record is
> still within the Producer buffer (waiting to be sent) the record would
> still be sent to the original leader from record creation time. And in this
> case the request.timeout.ms would apply to the send of that record
> failing.
>
> Only records created when send is called on the Producer after broker
> connection failure and re-fetch of the metadata will be sent to the new
> leader.
>
> Could you please confirm my understanding is correct?
>
> Thanks,
> Mark
>


-- 
-Regards,
Mayuresh R. Gharat
(862) 250-7125