[DISCUSS] KIP-1339: Make KafkaConfigBackingStore read_write_total_timeout configurable

[Henry Haiying Cai via dev <de...@kafka.apache.org>]

I would like to start discussing KIP-1339: Make KafkaConfigBackingStore read_write_total_timeout configurable

KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP-1339%3A+Make+KafkaConfigBackingStore+read_write_total_timeout+configurable

In Kafka Connect, the KafkaConfigBackingStore class is responsible for persisting connector and task configurations in a Kafka topic. Currently, the timeout for synchronous write operations and then reading back from the configuration topic is hardcoded via the READ_WRITE_TOTAL_TIMEOUT_MS constant, set to 30,000 milliseconds (30 seconds).

While 30 seconds is sufficient for many environments, this hardcoded limit can be problematic in some specific scenarios.

There were earlier discussions (https://lists.apache.org/thread/flpm1qrm7cwj6fqr1j8c0p8sbqysgv34) on extending the timeout value. This KIP proposes making this timeout value a configurable parameter.

Any feedback is appreciated.

Thanks and Regards

Henry Cai

Re: [DISCUSS] KIP-1339: Make KafkaConfigBackingStore read_write_total_timeout configurable

[Henry Haiying Cai via dev <de...@kafka.apache.org>]

Chris,

Thanks for the quick reply.  One thing to clarify from last time we talked to you.  We resolved the incident last month by your recommendation (increasing batch.size and linger.ms for the config topic's producer), but we worry about this issue might come back when we ramp more KC connector/tasks in the future.  Since it will take quite some time/effort to get the fix into Kafka upstream so we want to proactively address this issue by giving us another knob to turn during incident crunching time.

For the environment in our company, it is more than 500 tasks.  500 tasks is for one topic in that cluster but we have other topics also running through KC in that cluster and they need to share the same config topic.  The total number is more like 5000 tasks.  And we are planning to increase the parallelism more in the near future.

During the incident time, the consumer groups keeps rebalancing and there is new coordinator being elected very frequently and all tasks' config needs to be written to config topic very frequently due to those rebalances.  There are several round trip operations needs to finish with that hard-coded 30 second (configTopic consumer readToEnd, putting task config into configTopic for each task synchronously, readToEnd again, write commit message, readToEnd again), the consumer.readToEnd can also be long for a newly elected coordinator when there are many messages in the config topic accumulated in the topic due to quick consumer group rebalances (and the compact thread has not got the time to compact the topic).

We worried about that hard-coded 30 second ceiling in a large cluster during incident time when lots of consumer rebalance was happening, we want to have some more ceiling buffers.

Between the current hard-coded ceiling (30 seconds) and the default max.poll.timeout (300 seconds) for unhealthy worker detection, there seems some room we can tune.

For the suggestion you mentioned to allow tuning the linger.ms only for the config topic but not for the status topic, this is already there.  You can specify connect.config.storage.linger.ms to do exactly that.
 
For the suggestion that setting higher default of linger.ms for config topic, I am not sure whether that works for all users.  Some users with small cluster might prefer 0 linger.ms for shorter latency.

Re: [DISCUSS] KIP-1339: Make KafkaConfigBackingStore read_write_total_timeout configurable

[Chris Egerton <fe...@gmail.com>]

Hi Henry,

Thanks for getting the ball rolling on this. Funnily enough, I've just
started running into the same problem in the wild as I've been working on a
prototype Connect deployment for my company.

One thing that sticks out to me in the KIP is the call-out of 500 tasks.
That seems like a fairly low number to be hitting the 30-second timeout,
no? Less than 17 task configs per second.

Could we maybe think first about how to improve the performance of the
worker when producing to the config topic, instead of allowing for
increasingly-higher timeouts for publishing task configs? I've had some
success with tweaking linger time, batch size, etc. An alternative KIP
based on this approach could 1) introduce granular configuration for the
worker's internal producers (e.g., allow users to configure linger time for
the config topic but not for the status topic) and 2) introduce new
high-throughout defaults for the config topic (although we'd have to be
pretty conservative about setting something like linger time since it'd set
a lower bound on the time that most control plane operations would rate).

I'm worried about the availability of Connect's control plane if the
leader's herder tick thread is occupied for too long writing task configs,
and would like to address the root of this problem if possible.

Cheers,

Chris

On Thu, May 14, 2026, 03:13 Henry Haiying Cai via dev <de...@kafka.apache.org>
wrote:

> I would like to start discussing KIP-1339: Make KafkaConfigBackingStore
> read_write_total_timeout configurable
>
> KIP:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1339%3A+Make+KafkaConfigBackingStore+read_write_total_timeout+configurable
>
> In Kafka Connect, the KafkaConfigBackingStore class is responsible for
> persisting connector and task configurations in a Kafka topic. Currently,
> the timeout for synchronous write operations and then reading back from the
> configuration topic is hardcoded via the READ_WRITE_TOTAL_TIMEOUT_MS
> constant, set to 30,000 milliseconds (30 seconds).
>
> While 30 seconds is sufficient for many environments, this hardcoded limit
> can be problematic in some specific scenarios.
>
> There were earlier discussions (
> https://lists.apache.org/thread/flpm1qrm7cwj6fqr1j8c0p8sbqysgv34) on
> extending the timeout value. This KIP proposes making this timeout value a
> configurable parameter.
>
> Any feedback is appreciated.
>
> Thanks and Regards
>
> Henry Cai
>