[jira] [Created] (KAFKA-21338) Use exponential backoff with jitter for internal topic retries in InternalTopicManager

["Bill Bejeck (Jira)" <ji...@apache.org>]

Bill Bejeck created KAFKA-21338:
-----------------------------------

             Summary: Use exponential backoff with jitter for internal topic retries in InternalTopicManager
                 Key: KAFKA-21338
                 URL: https://issues.apache.org/jira/browse/KAFKA-21338
             Project: Kafka
          Issue Type: Bug
          Components: streams
            Reporter: Bill Bejeck
             Fix For: 4.5.0


In Kafka Streams the InternalTopicManager retries internal topic
describe/create/validate on a fixed \{{retry.backoff.ms}} (default 100 ms) until
{\{max.poll.interval.ms}} / 2 (default 150 s). The backoff never grows and has no jitter. During a
broker outage that is up to ~1,500 retry rounds per application, and applications on the same
cluster that rebalance after the same outage retry in lockstep.

The producer, consumer and admin clients already back off exponentially with jitter (KIP-580,
{\{org.apache.kafka.common.utils.ExponentialBackoff}}). InternalTopicManager should do the same.

Out of scope: the streams rebalance protocol (KIP-1071), where the broker creates internal topics.

Depends on KAFKA-21033 (https://github.com/apache/kafka/pull/23363), which moves the backoff back
to one sleep per round in maybeSleep and switches it to time.sleep. This change should build on
that.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)