[jira] [Resolved] (KAFKA-19160) Fetching stable offsets is very slow with a large number of partition and open transactions

["David Jacot (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-19160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

David Jacot resolved KAFKA-19160.
---------------------------------
    Fix Version/s: 4.0.1
                   4.1.0
       Resolution: Fixed

> Fetching stable offsets is very slow with a large number of partition and open transactions
> -------------------------------------------------------------------------------------------
>
>                 Key: KAFKA-19160
>                 URL: https://issues.apache.org/jira/browse/KAFKA-19160
>             Project: Kafka
>          Issue Type: Bug
>          Components: group-coordinator
>            Reporter: Sean Quah
>            Assignee: Sean Quah
>            Priority: Major
>             Fix For: 4.0.1, 4.1.0
>
>
> When fetching stable offsets in the group coordinator, we iterate over all requested partitions and for each partition, check if any ongoing transactions have updated the group's offset for that partition. This is O(#partitions * #open_transactions).
> For a large topic with 4,000 partitions and a correspondingly large group, we might have up to 4,000 open transactions. Fetching stable offsets for all partitions in this case can take around 350 ms.
> It looks like the old coordinator also has this issue but it's being removed soon anyway.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)