You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Philip Nee (Jira)" <ji...@apache.org> on 2024/12/30 17:38:00 UTC

[jira] [Created] (KAFKA-18376) High CPU load when AsyncKafkaConsumer uses a small max poll value

Philip Nee created KAFKA-18376:
----------------------------------

             Summary: High CPU load when AsyncKafkaConsumer uses a small max poll value
                 Key: KAFKA-18376
                 URL: https://issues.apache.org/jira/browse/KAFKA-18376
             Project: Kafka
          Issue Type: Bug
          Components: clients, consumer
            Reporter: Philip Nee
            Assignee: Philip Nee


We stress tested the AsyncConsumer using maxPoll = 5 and observed abnormally high CPU usage.  Under normal usage (with defaults), the consumers on average use around 10% of the CPU with 20mb/s byte rate, which is aligned with the ClassicKafkaConsumer.  As we tested the consumer with a small max poll value, we observed the CPU usage spikes to > 50% while the classic consumer stays at around 10%.

 

_note: percentage of CPU usage may depend on the running pod hardware._

 

The profiling results shows two major contributors to the CPU cycles
 # AsyncKafkaConsumer.updateFetchPosition (addAndGet & new CheckAndUpdatePositionEvent())
 # AbstractFetch.fetchablePartitions from the fetchrequestmanager

 

for AsyncKafkaConsumer.updateFetchPosition - It seems like
 * UUID generation can become quite expensive. This is particularly noticeable when creating large number of events
 * ConsumerUtils.getResult, which uses future.get() also consumes quite a bit of CPU cycles

for fetchablePartitions, FetchBuffer.bufferedPartitions which uses Java ConcurrentLinkedQueue.forEach also consumes quite a bit of CPUs.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)