You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Philip Nee (Jira)" <ji...@apache.org> on 2024/12/30 17:38:00 UTC
[jira] [Created] (KAFKA-18376) High CPU load when AsyncKafkaConsumer uses a small max poll value
Philip Nee created KAFKA-18376:
----------------------------------
Summary: High CPU load when AsyncKafkaConsumer uses a small max poll value
Key: KAFKA-18376
URL: https://issues.apache.org/jira/browse/KAFKA-18376
Project: Kafka
Issue Type: Bug
Components: clients, consumer
Reporter: Philip Nee
Assignee: Philip Nee
We stress tested the AsyncConsumer using maxPoll = 5 and observed abnormally high CPU usage. Under normal usage (with defaults), the consumers on average use around 10% of the CPU with 20mb/s byte rate, which is aligned with the ClassicKafkaConsumer. As we tested the consumer with a small max poll value, we observed the CPU usage spikes to > 50% while the classic consumer stays at around 10%.
_note: percentage of CPU usage may depend on the running pod hardware._
The profiling results shows two major contributors to the CPU cycles
# AsyncKafkaConsumer.updateFetchPosition (addAndGet & new CheckAndUpdatePositionEvent())
# AbstractFetch.fetchablePartitions from the fetchrequestmanager
for AsyncKafkaConsumer.updateFetchPosition - It seems like
* UUID generation can become quite expensive. This is particularly noticeable when creating large number of events
* ConsumerUtils.getResult, which uses future.get() also consumes quite a bit of CPU cycles
for fetchablePartitions, FetchBuffer.bufferedPartitions which uses Java ConcurrentLinkedQueue.forEach also consumes quite a bit of CPUs.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)