You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Ritvik Gupta (Jira)" <ji...@apache.org> on 2025/09/08 04:22:00 UTC
[jira] [Comment Edited] (KAFKA-19652) Consumer throughput drops by 10 times with Kafka v3.9.0 in ZK mode
[ https://issues.apache.org/jira/browse/KAFKA-19652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18018722#comment-18018722 ]
Ritvik Gupta edited comment on KAFKA-19652 at 9/8/25 4:21 AM:
--------------------------------------------------------------
Hi,
Updating here on behalf of [~gargsha] -
*Topic configs :*
{{cleanup.policy=compact,delete; retention.ms=1200000; min.insync.replicas=2}}
*Consumer configs :*
{{fetch.min.bytes=131072,fetch.max.wait.ms=500}}
( acks=1 for the Producer )
I have generally observed this case for when the topic has a high throughput producer(s) running on it with upwards of *590 MB/sec* ( *600 K msgs/sec* with *1000 b* message size each ). Can also simulate this case with running multiple producers on the topic to achieve close to 600 MB/sec.
In the above test scenario, the consumer throughput drops down to {*}< 90MB/sec{*}. This is for a 1 Producer + 1 Consumer test, which should be the ideal test scenario and was getting me upto *250+ MB/sec* consumer throughput in the old {*}v3.5.1 Kafka cluster{*}.
My best guess is some form of loss in *insync replicas* ( with the high throughput producers as *acks=1* ), as only the leader replica would stay in-sync in case of a high throughput on the topic. And if this can have a performance impact on the consumer in some way ??
To reason on this I had conducted the below test ( *min.insync.replicas=2* vs *min.insync.replicas=1* ) -
| |
|*test*|*Test Name*|*Observed message rate (K msgs/sec*|*Test description*|
|1| | | |
|Consumer|Throughput test run ( Java client ) - Min ISR = 2 - 1 producer & 1 consumer|*{color:#de350b}89.21{color}*|Test Configs = acks:1, partitions:1, rf:3, min-isr:2, producers:1, consumers:1, batch_size:256KB|
|Producer|Throughput test run ( Java client ) - Min ISR = 2 - 1 producer & 1 consumer|617.20|Test Configs = acks:1, partitions:1, rf:3, min-isr:2, producers:1, consumers:1, batch_size:256KB|
|2| | | |
|Consumer|Throughput test run ( Java client ) - Min ISR = 1 - 1 producer & 1 consumer|*{color:#00875a}298.99{color}*|Test Configs = acks:1, partitions:1, rf:3, min-isr:1, producers:1, consumers:1, batch_size:256KB|
|Producer|Throughput test run ( Java client ) - Min ISR = 1 - 1 producer & 1 consumer|597.20|Test Configs = acks:1, partitions:1, rf:3, min-isr:1, producers:1, consumers:1, batch_size:256KB|
Just to note I have also observed this heavy drop in consumer throughput in a *KRaft mode cluster ( v3.9.0 ).*
Let me know if to share more details regarding the tests I conducted or any other configs, for getting to debug this.
was (Author: JIRAUSER302370):
Hi,
Updating here on behalf of [~gargsha] -
*Topic configs :*
{{cleanup.policy=compact,delete; retention.ms=1200000; min.insync.replicas=2}}
*Consumer configs :*
{{fetch.min.bytes=131072,fetch.max.wait.ms=500}}
( acks=1 for the Producer )
I have generally observed this case for when the topic has a high throughput producer(s) running on it with upwards of *590 MB/sec* ( *600 K msgs/sec* with *1000 b* message size each ). Can also simulate this case with running multiple producers on the topic to achieve close to 600 MB/sec.
In the above test scenario, the consumer throughput drops down to {*}< 90MB/sec{*}. This is for a 1 Producer + 1 Consumer test, which should be the ideal test scenario and was getting me upto *250+ MB/sec* consumer throughput in the old {*}v3.5.1 Kafka cluster{*}.
My best guess is some form of loss in *insync replicas* ( with the high throughput producers as *acks=1* ), as only the leader replica would stay in-sync in case of a high throughput on the topic. And if this can have a performance impact on the consumer in some way ??
To reason on this I had conducted a below tests ( *min.insync.replicas=2* vs *min.insync.replicas=1* ) -
| |
|
|*test*|*Test Name*|*Observed message rate (K msgs/sec*|*Test description*|
|1| | | |
|Consumer|Throughput test run ( Java client ) - Min ISR = 2 - 1 producer & 1 consumer|*{color:#de350b}89.21{color}*|Test Configs = acks:1, partitions:1, rf:3, min-isr:2, producers:1, consumers:1, batch_size:256KB|
|Producer|Throughput test run ( Java client ) - Min ISR = 2 - 1 producer & 1 consumer|617.20|Test Configs = acks:1, partitions:1, rf:3, min-isr:2, producers:1, consumers:1, batch_size:256KB|
|2| | | |
|Consumer|Throughput test run ( Java client ) - Min ISR = 1 - 1 producer & 1 consumer|*{color:#00875a}298.99{color}*|Test Configs = acks:1, partitions:1, rf:3, min-isr:1, producers:1, consumers:1, batch_size:256KB|
|Producer|Throughput test run ( Java client ) - Min ISR = 1 - 1 producer & 1 consumer|597.20|Test Configs = acks:1, partitions:1, rf:3, min-isr:1, producers:1, consumers:1, batch_size:256KB|
|
Just to note I have also observed this heavy drop in consumer throughput in a *KRaft mode cluster ( v3.9.0 ).*
Let me know if to share more details regarding the tests I conducted or any other configs, for getting to debug this.
> Consumer throughput drops by 10 times with Kafka v3.9.0 in ZK mode
> ------------------------------------------------------------------
>
> Key: KAFKA-19652
> URL: https://issues.apache.org/jira/browse/KAFKA-19652
> Project: Kafka
> Issue Type: Bug
> Components: clients, consumer
> Affects Versions: 3.9.0
> Reporter: Sharad Garg
> Priority: Blocker
>
> Kafka consumer throughput in best-case drops by ~10 times after upgrading to kafka v3.9.0 from v3.5.1. Note that this is in ZK mode and KRAFT migration is not done yet.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)