You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Honghai Chen <Ho...@microsoft.com> on 2015/04/23 07:01:02 UTC
How to reduce produce-localtimems 999th percentile?
We got 300ms for produce-localtimems .999th percentile for some machines of one cluster, actually the value for other machines of the cluster at most around 100ms.
The QPS only 20~80 per second. The strange things is that only several machines has this kind of issue, all other machines has no this kind of issue. And the 99th percentile also less than 80ms. And the disk queue length less than 0.2
We're run on windows (after apply fix of kafka-1646) and we enable JBOD.
We dump the JMX counters every two minutes, here is some number:
1,04/23/2015 00:00:00, ,kafka.network.requestmetrics.produce-localtimems.999thpercentile,232.00
1,04/23/2015 00:02:00, ,kafka.network.requestmetrics.produce-localtimems.999thpercentile,33
1,04/23/2015 00:04:00, ,kafka.network.requestmetrics.produce-localtimems.999thpercentile,73
1,04/23/2015 00:06:00, ,kafka.network.requestmetrics.produce-localtimems.999thpercentile,98
1,04/23/2015 00:08:00, ,kafka.network.requestmetrics.produce-localtimems.999thpercentile,188.00
1,04/23/2015 00:10:00, ,kafka.network.requestmetrics.produce-localtimems.999thpercentile,93
1,04/23/2015 00:12:00, ,kafka.network.requestmetrics.produce-localtimems.999thpercentile,156.00
1,04/23/2015 00:14:00, kafka.network.requestmetrics.produce-localtimems.999thpercentile,300.00
1,04/23/2015 00:16:00, kafka.network.requestmetrics.produce-localtimems.999thpercentile,37
Thanks, Honghai
RE: How to reduce produce-localtimems 999th percentile?
Posted by Honghai Chen <Ho...@microsoft.com>.
We're using requireAck=0, we have tried sync/async producer, got same issue, any hint for more investigation?
Thanks, Honghai Chen
-----Original Message-----
From: Honghai Chen [mailto:Honghai.Chen@microsoft.com]
Sent: Thursday, April 23, 2015 1:01 PM
To: users@kafka.apache.org
Subject: How to reduce produce-localtimems 999th percentile?
We got 300ms for produce-localtimems .999th percentile for some machines of one cluster, actually the value for other machines of the cluster at most around 100ms.
The QPS only 20~80 per second. The strange things is that only several machines has this kind of issue, all other machines has no this kind of issue. And the 99th percentile also less than 80ms. And the disk queue length less than 0.2
We're run on windows (after apply fix of kafka-1646) and we enable JBOD.
We dump the JMX counters every two minutes, here is some number:
1,04/23/2015 00:00:00, ,kafka.network.requestmetrics.produce-localtimems.999thpercentile,232.00
1,04/23/2015 00:02:00, ,kafka.network.requestmetrics.produce-localtimems.999thpercentile,33
1,04/23/2015 00:04:00, ,kafka.network.requestmetrics.produce-localtimems.999thpercentile,73
1,04/23/2015 00:06:00, ,kafka.network.requestmetrics.produce-localtimems.999thpercentile,98
1,04/23/2015 00:08:00, ,kafka.network.requestmetrics.produce-localtimems.999thpercentile,188.00
1,04/23/2015 00:10:00, ,kafka.network.requestmetrics.produce-localtimems.999thpercentile,93
1,04/23/2015 00:12:00, ,kafka.network.requestmetrics.produce-localtimems.999thpercentile,156.00
1,04/23/2015 00:14:00, kafka.network.requestmetrics.produce-localtimems.999thpercentile,300.00
1,04/23/2015 00:16:00, kafka.network.requestmetrics.produce-localtimems.999thpercentile,37
Thanks, Honghai