You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Sean Quah (Jira)" <ji...@apache.org> on 2025/04/16 11:55:00 UTC

[jira] [Created] (KAFKA-19160) Fetching stable offsets is very slow with a large number of partition and open transactions

Sean Quah created KAFKA-19160:
---------------------------------

             Summary: Fetching stable offsets is very slow with a large number of partition and open transactions
                 Key: KAFKA-19160
                 URL: https://issues.apache.org/jira/browse/KAFKA-19160
             Project: Kafka
          Issue Type: Bug
          Components: group-coordinator
            Reporter: Sean Quah
            Assignee: Sean Quah


When fetching stable offsets in the group coordinator, we iterate over all requested partitions and for each partition, check if any ongoing transactions have updated the group's offset for that partition. This is O(#partitions * #open_transactions).

For a large topic with 4,000 partitions and a correspondingly large group, we might have up to 4,000 open transactions. Fetching stable offsets for all partitions in this case can take around 350 ms.

It looks like the old coordinator also has this issue but it's being removed soon anyway.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)