[jira] [Created] (KAFKA-17751) Contoller high CPU when formatted with --initial-controllers

["Juha Mynttinen (Jira)" <ji...@apache.org>]

Juha Mynttinen created KAFKA-17751:
--------------------------------------

             Summary: Contoller high CPU when formatted with --initial-controllers 
                 Key: KAFKA-17751
                 URL: https://issues.apache.org/jira/browse/KAFKA-17751
             Project: Kafka
          Issue Type: Bug
    Affects Versions: 3.9.0
            Reporter: Juha Mynttinen
         Attachments: Screenshot 2024-10-09 at 9.15.06.png, c1.properties, c2.properties, c3.properties, c4.properties

Hey,

I'm using 3.9.0 RC0.

I noticed that formatting a simple three node controller cluster with --initial-controllers and starting the controller leads to a situation where the non-leader voters consume a lot of CPU.

Here are the steps to reproduce. The needed configuration files are attached.



Clean up and setup the environment.

rm -rf /tmp/controllers && \
mkdir -p /tmp/controllers/c1 && \
mkdir -p /tmp/controllers/c2 && \
mkdir -p /tmp/controllers/c3 && \
mkdir -p /tmp/controllers/c4

export KAFKA_HOME=<your_kafka_3_9_home>

Format the controllers

$KAFKA_HOME/bin/kafka-storage.sh format --cluster-id 00000000-0000-0000-0000-000000000001 --initial-controllers 1001@localhost:10001:AAAAAAAAAAEAAAAAAAAAAA,1002@localhost:10002:AAAAAAAAAAEAAAAAAAAAAA,1003@localhost:10003:AAAAAAAAAAEAAAAAAAAAAA --config c1.properties
$KAFKA_HOME/bin/kafka-storage.sh format --cluster-id 00000000-0000-0000-0000-000000000001 --initial-controllers 1001@localhost:10001:AAAAAAAAAAEAAAAAAAAAAA,1002@localhost:10002:AAAAAAAAAAEAAAAAAAAAAA,1003@localhost:10003:AAAAAAAAAAEAAAAAAAAAAA --config c2.properties
$KAFKA_HOME/bin/kafka-storage.sh format --cluster-id 00000000-0000-0000-0000-000000000001 --initial-controllers 1001@localhost:10001:AAAAAAAAAAEAAAAAAAAAAA,1002@localhost:10002:AAAAAAAAAAEAAAAAAAAAAA,1003@localhost:10003:AAAAAAAAAAEAAAAAAAAAAA --config c3.properties

Start the controllers, in separate terminals

$KAFKA_HOME/bin/kafka-run-class.sh -name kafkaService kafka.Kafka c1.properties
$KAFKA_HOME/bin/kafka-run-class.sh -name kafkaService kafka.Kafka c2.properties
$KAFKA_HOME/bin/kafka-run-class.sh -name kafkaService kafka.Kafka c3.properties

Observe two of the controllers have CPU usage at 100%. If you check which PID is which, you can see that it's the two processes that are voters that have elevated CPU. The CPU usage of the leader is fine.

I did in an slightly different environment some profiling. The screenshot is attached.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)