[jira] [Created] (KAFKA-15178) Poor performance of ConsumerCoordinator with many TopicPartitions

["Nicholas Telford (Jira)" <ji...@apache.org>]

Nicholas Telford created KAFKA-15178:
----------------------------------------

             Summary: Poor performance of ConsumerCoordinator with many TopicPartitions
                 Key: KAFKA-15178
                 URL: https://issues.apache.org/jira/browse/KAFKA-15178
             Project: Kafka
          Issue Type: Bug
          Components: consumer
    Affects Versions: 3.5.0
            Reporter: Nicholas Telford
            Assignee: Nicholas Telford
         Attachments: pollPhase.png

Doing some profiling of my Kafka Streams application, I noticed that the {{pollPhase}} suffers from a minor performance issue.

See flame graph below.  !pollPhase.png|width=1028,height=308!

{{ConsumerCoordinator.poll}} calls {{{}rejoinNeededOrPending{}}}, which checks the current {{metadataSnapshot}} against the {{{}assignmentSnapshot{}}}. This comparison is a deep-equality check, and if there's a large number of topic-partitions being consumed by the application, then this comparison can perform poorly.

I suspect this can be trivially addressed with a {{boolean}} flag that indicates when the {{metadataSnapshot}} has been updated (or is "dirty"), and actually needs to be checked, since most of the time it should be identical to {{{}assignmentSnapshot{}}}.

I plan to raise a PR with this optimization to address this issue.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)