You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Chris Pettitt (Jira)" <ji...@apache.org> on 2019/11/14 20:19:00 UTC
[jira] [Updated] (KAFKA-9191) Kafka throughput suffers
substantially when scaling topic partitions with small messages
[ https://issues.apache.org/jira/browse/KAFKA-9191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Pettitt updated KAFKA-9191:
---------------------------------
Description:
We have observed, using two entirely different tools, that a simple Kafka application (read 1 topic and immediately produce to another) suffers substantial performance degradation when scaling up topics. Below is the output of one of these tools, showing that going from 1 partition to 1000 partitions yields a ~30% throughput decrease when messages are 100 bytes long.
Using the same two tools, we observed that increasing the message size to 512 bytes yields a throughput increase of ~20% going from 1 topic partition to 1000 topic partitions with all other variables held constant.
|Kafka Core Testing| | | | | | | | | | |
|Enable Transaction|Batch Size (b)|Linger (ms)|Max Inflight|Commit Interval (ms)|Num Records|Record Size (b)|Num Input Topics|1 Partition MB/s|1000 Partitions MB/s|MB/s delta|
|FALSE|16384|100|5|1000|20000000|100|1|45.633625|31.482193|-31.01%|
|FALSE|16384|100|5|1000|20000000|512|1|70.217902|85.319107|21.51%|
was:
We have observed, using two entirely different tools, that a simple Kafka application (read 1 topic and immediately produce to another) suffers substantial performance degradation when scaling up topics. Below is the output of one of these tools, showing that going from 1 partition to 1000 partitions yields a ~30% throughput decrease when messages are 100 bytes long.
Using the same two tools, we observed that increasing the message size to 512 bytes yields a throughput increase of ~20% going from 1 topic partition to 1000 topic partitions with all other variables held constant.
|Kafka Core Testing| | | | | | | | | | |
|Enable Transaction|Batch Size (b)|Linger (ms)|Max Inflight|Commit Interval (ms)|Num Records|Record Size (b)|# Input Topics|1 Partition MB/s|1000 Partitions MB/s|MB/s delta|
|FALSE|16384|100|5|1000|20000000|100|1|45.633625|31.482193|-31.01%|
|FALSE|16384|100|5|1000|20000000|512|1|70.217902|85.319107|21.51%|
> Kafka throughput suffers substantially when scaling topic partitions with small messages
> ----------------------------------------------------------------------------------------
>
> Key: KAFKA-9191
> URL: https://issues.apache.org/jira/browse/KAFKA-9191
> Project: Kafka
> Issue Type: Bug
> Reporter: Chris Pettitt
> Priority: Major
>
> We have observed, using two entirely different tools, that a simple Kafka application (read 1 topic and immediately produce to another) suffers substantial performance degradation when scaling up topics. Below is the output of one of these tools, showing that going from 1 partition to 1000 partitions yields a ~30% throughput decrease when messages are 100 bytes long.
> Using the same two tools, we observed that increasing the message size to 512 bytes yields a throughput increase of ~20% going from 1 topic partition to 1000 topic partitions with all other variables held constant.
>
> |Kafka Core Testing| | | | | | | | | | |
> |Enable Transaction|Batch Size (b)|Linger (ms)|Max Inflight|Commit Interval (ms)|Num Records|Record Size (b)|Num Input Topics|1 Partition MB/s|1000 Partitions MB/s|MB/s delta|
> |FALSE|16384|100|5|1000|20000000|100|1|45.633625|31.482193|-31.01%|
> |FALSE|16384|100|5|1000|20000000|512|1|70.217902|85.319107|21.51%|
--
This message was sent by Atlassian Jira
(v8.3.4#803005)