You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by 牟忠玉 <mu...@yeah.net> on 2023/02/20 16:10:22 UTC

Kafka partition problem

Why is Kafka's QPS performance very poor with a large number of partitions (such as 10,000 partitions)? What is the reason for this? Do you have any plans to optimize this in the future? Or are there already optimizations in newer versions?

Re: Kafka partition problem

Posted by "Brebner, Paul" <Pa...@netapp.com.INVALID>.
Hi, I have a few blogs on Kafka partitions performance for background if you are interested:


  1.  https://www.instaclustr.com/blog/the-power-of-kafka-partitions-how-to-get-the-most-out-of-your-kafka-cluster/
The Power of Apache Kafka® Partitions: How to Get the Most out of Your Kafka Cluster<https://www.instaclustr.com/blog/the-power-of-kafka-partitions-how-to-get-the-most-out-of-your-kafka-cluster/>
Partitions are the main concurrency mechanism in Kafka. Learn how to derive maximum benefit by using partitions for your kafka cluster.
www.instaclustr.com


  2.  New 3 part KRaft series: https://www.instaclustr.com/blog/apache-kafka-kraft-abandons-the-zookeeper-part-1-partitions-and-data-performance/
  3.  ApacheCon NA 2022 talk https://apachecon.com/acna2022/slides/04_Brebner_The_Impact_Of.pdf

Basically throughput with increasing partitions tends to drop due to replication overhead, but newer versions are better. The "problem" with KRaft mode may be that it's too easy to create topics with lots of partitions, with potential performance implications still.

Regards, Paul
________________________________
From: 牟忠玉 <mu...@yeah.net>
Sent: 21 February 2023 03:10 AM
To: users@kafka.apache.org <us...@kafka.apache.org>
Subject: Kafka partition problem

NetApp Security WARNING: This is an external email. Do not click links or open attachments unless you recognize the sender and know the content is safe.




Why is Kafka's QPS performance very poor with a large number of partitions (such as 10,000 partitions)? What is the reason for this? Do you have any plans to optimize this in the future? Or are there already optimizations in newer versions?