Is it possible to succeed message delivery in case TimeoutException thrown (by delivery.timeout.ms)

[김광용 <ba...@gmail.com>]

Hi, I am newbie in kafka.

I am studying kafka properties to build robust applications.
When I look at Kafka producer properties, I was curious if it is possible
that a client receives timeout exception (by delivery.timeout.ms) but
succeed message delivery.

KIP-91 describes that situation.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-91+Provide+Intuitive+User+Timeouts+in+The+Producer

The "timer" for each batch starts "ticking" at the creation of the batch.
Batches expire in order when max.in.flight.request.per.connection==1. An
in-flight batch expire when delivery.timeout.ms has passed since the batch
creation irrespective of whether the batch is in flight or not. However,
the producer waits the full request.timeout.ms for the in-flight request. *This
implies **that user might be notified of batch expiry while a batch is
still in-flight.*

Could anyone explain about this?

Thank you in advance.

Re: Is it possible to succeed message delivery in case TimeoutException thrown (by delivery.timeout.ms)

[김광용 <ba...@gmail.com>]

Thank you!

I was wondering if the possibility exists.
If the possibility depends on producer `ack` config, when I set `ack=all`
and follow default `delivery.timeout.ms=120000`, `request.timeout.ms=30000`,
`linger.ms=0`, `replica.lag.time.max.ms=30000` configs, *the possibility* can
be removed completely? or just reduced?

I would like to reduce the possibility as low as possible that producer
resends messages which are already sent to broker successfully.

-- Thank you

2020년 7월 21일 (화) 오전 12:16, Ricardo Ferreira <ri...@riferrei.com>님이 작성:

> This possibility highly depends of the option used for the `ack` property.
> If you set to `0` then there is high probability. If you set to `1` then
> probability decreases, and so forth so on. But in general, you can minimize
> the possibility of this to happen using the following timing configuration:
>
> delivery.timeout.ms > request.timeout.ms + linger.ms >
> replica.lag.time.max.ms
>
> -- Ricardo
> On 7/19/20 10:49 PM, 김광용 wrote:
>
> Hi, I am newbie in kafka.
>
> I am studying kafka properties to build robust applications.
> When I look at Kafka producer properties, I was curious if it is possible
> that a client receives timeout exception (by delivery.timeout.ms) but
> succeed message delivery.
>
> KIP-91 describes that situation.https://cwiki.apache.org/confluence/display/KAFKA/KIP-91+Provide+Intuitive+User+Timeouts+in+The+Producer
>
> The "timer" for each batch starts "ticking" at the creation of the batch.
> Batches expire in order when max.in.flight.request.per.connection==1. An
> in-flight batch expire when delivery.timeout.ms has passed since the batch
> creation irrespective of whether the batch is in flight or not. However,
> the producer waits the full request.timeout.ms for the in-flight request. *This
> implies **that user might be notified of batch expiry while a batch is
> still in-flight.*
>
> Could anyone explain about this?
>
> Thank you in advance.
>
>
>

Re: Is it possible to succeed message delivery in case TimeoutException thrown (by delivery.timeout.ms)

[Ricardo Ferreira <ri...@riferrei.com>]

This possibility highly depends of the option used for the `ack` 
property. If you set to `0` then there is high probability. If you set 
to `1` then probability decreases, and so forth so on. But in general, 
you can minimize the possibility of this to happen using the following 
timing configuration:

delivery.timeout.ms > request.timeout.ms + linger.ms > 
replica.lag.time.max.ms

-- Ricardo

On 7/19/20 10:49 PM, 김광용 wrote:
> Hi, I am newbie in kafka.
>
> I am studying kafka properties to build robust applications.
> When I look at Kafka producer properties, I was curious if it is possible
> that a client receives timeout exception (by delivery.timeout.ms) but
> succeed message delivery.
>
> KIP-91 describes that situation.
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-91+Provide+Intuitive+User+Timeouts+in+The+Producer
>
> The "timer" for each batch starts "ticking" at the creation of the batch.
> Batches expire in order when max.in.flight.request.per.connection==1. An
> in-flight batch expire when delivery.timeout.ms has passed since the batch
> creation irrespective of whether the batch is in flight or not. However,
> the producer waits the full request.timeout.ms for the in-flight request. *This
> implies **that user might be notified of batch expiry while a batch is
> still in-flight.*
>
> Could anyone explain about this?
>
> Thank you in advance.
>