[jira] [Created] (KAFKA-5851) CPU overhead to having partitions (even inactive)

["Stephane Maarek (JIRA)" <ji...@apache.org>]

Stephane Maarek created KAFKA-5851:
--------------------------------------

             Summary: CPU overhead to having partitions (even inactive)
                 Key: KAFKA-5851
                 URL: https://issues.apache.org/jira/browse/KAFKA-5851
             Project: Kafka
          Issue Type: Bug
         Environment: 0.11.0.0, 0.10.x
            Reporter: Stephane Maarek


We're running on three r4.xlarge, and this is our dev environment.

Some people managed to create 1200 topics with 3 partitions, so we end up at 4000 partitions per broker (we have a replication factor of 3). Even though no data goes through the cluster (it sits at a comfortable 20 messages per seconds), we saw CPU at 100% (out of 400%).

I went ahead today and deleted 700 topics that I knew were unused. And CPU went drastically down. See image

!https://i.imgur.com/OIPTwDM.png!

We use the defaults for our brokers, and they use PLAINTEXT internally to replicate. 

I'm not sure of the root cause of this, threads, replication, log cleanup, etc, and I guess it wouldn't be too hard to replicate (just create 1000 topics on a vanilla cluster and see CPU go up). 

Hope that helps



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)