You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Kirk True (Jira)" <ji...@apache.org> on 2025/02/25 20:06:00 UTC
[jira] [Assigned] (KAFKA-18376) High CPU load when AsyncKafkaConsumer uses a small max poll value
[ https://issues.apache.org/jira/browse/KAFKA-18376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kirk True reassigned KAFKA-18376:
---------------------------------
Assignee: Kirk True (was: Philip Nee)
> High CPU load when AsyncKafkaConsumer uses a small max poll value
> -----------------------------------------------------------------
>
> Key: KAFKA-18376
> URL: https://issues.apache.org/jira/browse/KAFKA-18376
> Project: Kafka
> Issue Type: Bug
> Components: clients, consumer
> Reporter: Philip Nee
> Assignee: Kirk True
> Priority: Major
> Labels: consumer-threading-refactor, performance
> Fix For: 4.1.0
>
>
> We stress tested the AsyncConsumer using maxPoll = 5 and observed abnormally high CPU usage. Under normal usage (with defaults), the consumers on average use around 10% of the CPU with 20mb/s byte rate, which is aligned with the ClassicKafkaConsumer. As we tested the consumer with a small max poll value, we observed the CPU usage spikes to > 50% while the classic consumer stays at around 10%.
>
> _note: percentage of CPU usage may depend on the running pod hardware._
>
> The profiling results shows two major contributors to the CPU cycles
> # AsyncKafkaConsumer.updateFetchPosition (addAndGet & new CheckAndUpdatePositionEvent())
> # AbstractFetch.fetchablePartitions from the fetchrequestmanager
>
> for AsyncKafkaConsumer.updateFetchPosition - It seems like
> * UUID generation can become quite expensive. This is particularly noticeable when creating large number of events
> * ConsumerUtils.getResult, which uses future.get() also consumes quite a bit of CPU cycles
> for fetchablePartitions, FetchBuffer.bufferedPartitions which uses Java ConcurrentLinkedQueue.forEach also consumes quite a bit of CPUs.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)