Kafka Connect: producer idempotence after increasing producer.max.in.flight.requests.per.connection

[Yong Gang <ch...@gmail.com>]

Hi guys,

We are using Kafka Connect to do Kafka to Kafka data replication between
DCs.
For this config item *producer.max.in.flight.requests.per.connection* the
default value in KafkaProducer (Kafka client) is 5 and can still
maintain idempotence, but in Kafka Connect the default value of this is 1.

I wonder if increasing *producer.max.in.flight.requests.per.connection *to
5 can Kafka Connect still be idempotent?

Thanks,
YongGang

Re: Kafka Connect: producer idempotence after increasing producer.max.in.flight.requests.per.connection

[Liam Clarke-Hutchinson <li...@adscale.co.nz>]

Hi Yong Gang,

Idempotence is not guaranteed if you increase max in flight requests.
Because if 5 requests are dispatched, but 2 requests fail and are retried,
the messages in those 2 requests will arrive after the messages in the
other three requests, so ordering will be changed.

Assuming everything is happy in your cluster, network etc., you won't see
immediate negative effects on idempotence when increasing that property.

But when the cluster/network/etc. is heavily loaded, then you'll start to
see the negative effects.

Does that make sense?

Kind regards,

Liam Clarke-Hutchinson

On Fri, 23 Apr. 2021, 11:44 am Yong Gang, <ch...@gmail.com> wrote:

> Hi guys,
>
> We are using Kafka Connect to do Kafka to Kafka data replication between
> DCs.
> For this config item *producer.max.in.flight.requests.per.connection* the
> default value in KafkaProducer (Kafka client) is 5 and can still
> maintain idempotence, but in Kafka Connect the default value of this is 1.
>
> I wonder if increasing *producer.max.in.flight.requests.per.connection *to
> 5 can Kafka Connect still be idempotent?
>
> Thanks,
> YongGang
>