[jira] [Created] (KAFKA-9953) support multiple consumerGroupCoordinators in TransactionManager

["Joost van de Wijgerd (Jira)" <ji...@apache.org>]

Joost van de Wijgerd created KAFKA-9953:
-------------------------------------------

             Summary: support multiple consumerGroupCoordinators in TransactionManager
                 Key: KAFKA-9953
                 URL: https://issues.apache.org/jira/browse/KAFKA-9953
             Project: Kafka
          Issue Type: Improvement
          Components: clients
    Affects Versions: 2.5.0
            Reporter: Joost van de Wijgerd


We are using kafka with a transactional producer and have the following use case:

3 KafkaConsumers (each with their own ConsumerGroup) polled by the same thread and 1 transactional kafka producer. When we add the offsets to the transaction we run into the following problem: 

TransactionManager only keeps track of 1 consumerGroupCoordinator, however it can be that some consumerGroupCoordinators are on another node, now we constantly see the TransactionManager switching between nodes, this has overhead of 1 failing _TxnOffsetCommitRequest_ and 1 unnecessary _FindCoordinatorRequest_.

Also with  _retry.backoff.ms_ set to 100 by default this is causing a pause of 100ms for every other transaction (depending on what KafkaConsumer triggered the transaction of course)

If the TransactionManager could keep track of coordinator nodes per consumerGroupId this problem would be solved. 

I have already a patch for this but still need to test it. Will add it to the ticket when that is done



--
This message was sent by Atlassian Jira
(v8.3.4#803005)