You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Kafka <ka...@126.com> on 2016/06/25 07:14:48 UTC
delay of producer and consumer in kafka 0.9 is too big to be accepted
Hi all，
my kafka cluster is composed of three brokers with each have 8core cpu and 8g memory and 1g network card.
with java async client，I sent 1000000 messages with size of 1024 bytes per message ,the send gap between each sending is 20us，the consumer’s config is like this，fetch.min.bytes is set to 1, fetch.wait.max.ms is set to 100.
to avoid the inconformity bewteen two machines，I start producer and consumer at the same machine，the machine’s configurations have enough resources to satisfy these two clients.
I start consumer before producer on each test，with the sending timestamp in each message，when consumer receive the message，then I can got the consumer delay through the substraction between current timesstamp and sending timestamp.
when I set acks to 0，replica to 2，then the average producer delay is 2.98ms， the average consumer delay is 52.23ms.
when I set acks to 1，replica to 2，then the average producer delay is 3.9ms，the average consumer delay is 44.88ms.
when I set acks to -1, replica to 2, then the average producer delay is 1782ms, the average consumer delay is 1786ms.
I have two doubts，the first is why my consumer's delay with acks settled to 0 is logger than the consumer delay witch acks settled to 1.
the second is why the delay of producer and consumer is so big when I set acks to -1，I think this delay is can not be accepted.
and I found this delay is amplified with sending more messages.
any feedback is appreciated.
thanks
Re: delay of producer and consumer in kafka 0.9 is too big to be accepted
Posted by Kafka <ka...@126.com>.
can someone please explain why latency is so big for me？thanks
> 在 2016年6月25日，下午11:16，Jay Kreps <ja...@confluent.io> 写道：
>
> Can you sanity check this with the end-to-end latency test that ships with
> Kafka in the tools package?
>
> https://apache.googlesource.com/kafka/+/1769642bb779921267bd57d3d338591dbdf33842/core/src/main/scala/kafka/tools/TestEndToEndLatency.scala
>
> On Saturday, June 25, 2016, Kafka <ka...@126.com> wrote:
>
>> Hi all，
>> my kafka cluster is composed of three brokers with each have 8core
>> cpu and 8g memory and 1g network card.
>> with java async client，I sent 1000000 messages with size of 1024
>> bytes per message ,the send gap between each sending is 20us，the consumer’s
>> config is like this，fetch.min.bytes is set to 1, fetch.wait.max.ms is set
>> to 100.
>> to avoid the inconformity bewteen two machines，I start producer
>> and consumer at the same machine，the machine’s configurations have enough
>> resources to satisfy these two clients.
>>
>> I start consumer before producer on each test，with the sending
>> timestamp in each message，when consumer receive the message，then I can got
>> the consumer delay through the substraction between current timesstamp and
>> sending timestamp.
>> when I set acks to 0，replica to 2，then the average producer delay
>> is 2.98ms， the average consumer delay is 52.23ms.
>> when I set acks to 1，replica to 2，then the average producer delay
>> is 3.9ms，the average consumer delay is 44.88ms.
>> when I set acks to -1, replica to 2, then the average producer
>> delay is 1782ms, the average consumer delay is 1786ms.
>>
>> I have two doubts，the first is why my consumer's delay with acks
>> settled to 0 is logger than the consumer delay witch acks settled to 1.
>> the second is why the delay of producer and consumer is so big when I set
>> acks to -1，I think this delay is can not be accepted.
>> and I found this delay is amplified with sending more messages.
>>
>> any feedback is appreciated.
>> thanks
>>
>>
>>
>>
Re: delay of producer and consumer in kafka 0.9 is too big to be accepted
Posted by Kafka <ka...@126.com>.
can someone please explain why latency is so big for me？thanks
> 在 2016年6月25日，下午11:16，Jay Kreps <ja...@confluent.io> 写道：
>
> Can you sanity check this with the end-to-end latency test that ships with
> Kafka in the tools package?
>
> https://apache.googlesource.com/kafka/+/1769642bb779921267bd57d3d338591dbdf33842/core/src/main/scala/kafka/tools/TestEndToEndLatency.scala
>
> On Saturday, June 25, 2016, Kafka <ka...@126.com> wrote:
>
>> Hi all，
>> my kafka cluster is composed of three brokers with each have 8core
>> cpu and 8g memory and 1g network card.
>> with java async client，I sent 1000000 messages with size of 1024
>> bytes per message ,the send gap between each sending is 20us，the consumer’s
>> config is like this，fetch.min.bytes is set to 1, fetch.wait.max.ms is set
>> to 100.
>> to avoid the inconformity bewteen two machines，I start producer
>> and consumer at the same machine，the machine’s configurations have enough
>> resources to satisfy these two clients.
>>
>> I start consumer before producer on each test，with the sending
>> timestamp in each message，when consumer receive the message，then I can got
>> the consumer delay through the substraction between current timesstamp and
>> sending timestamp.
>> when I set acks to 0，replica to 2，then the average producer delay
>> is 2.98ms， the average consumer delay is 52.23ms.
>> when I set acks to 1，replica to 2，then the average producer delay
>> is 3.9ms，the average consumer delay is 44.88ms.
>> when I set acks to -1, replica to 2, then the average producer
>> delay is 1782ms, the average consumer delay is 1786ms.
>>
>> I have two doubts，the first is why my consumer's delay with acks
>> settled to 0 is logger than the consumer delay witch acks settled to 1.
>> the second is why the delay of producer and consumer is so big when I set
>> acks to -1，I think this delay is can not be accepted.
>> and I found this delay is amplified with sending more messages.
>>
>> any feedback is appreciated.
>> thanks
>>
>>
>>
>>
Re: delay of producer and consumer in kafka 0.9 is too big to be accepted
Posted by Jay Kreps <ja...@confluent.io>.
Can you sanity check this with the end-to-end latency test that ships with
Kafka in the tools package?
https://apache.googlesource.com/kafka/+/1769642bb779921267bd57d3d338591dbdf33842/core/src/main/scala/kafka/tools/TestEndToEndLatency.scala
On Saturday, June 25, 2016, Kafka <ka...@126.com> wrote:
> Hi all，
> my kafka cluster is composed of three brokers with each have 8core
> cpu and 8g memory and 1g network card.
> with java async client，I sent 1000000 messages with size of 1024
> bytes per message ,the send gap between each sending is 20us，the consumer’s
> config is like this，fetch.min.bytes is set to 1, fetch.wait.max.ms is set
> to 100.
> to avoid the inconformity bewteen two machines，I start producer
> and consumer at the same machine，the machine’s configurations have enough
> resources to satisfy these two clients.
>
> I start consumer before producer on each test，with the sending
> timestamp in each message，when consumer receive the message，then I can got
> the consumer delay through the substraction between current timesstamp and
> sending timestamp.
> when I set acks to 0，replica to 2，then the average producer delay
> is 2.98ms， the average consumer delay is 52.23ms.
> when I set acks to 1，replica to 2，then the average producer delay
> is 3.9ms，the average consumer delay is 44.88ms.
> when I set acks to -1, replica to 2, then the average producer
> delay is 1782ms, the average consumer delay is 1786ms.
>
> I have two doubts，the first is why my consumer's delay with acks
> settled to 0 is logger than the consumer delay witch acks settled to 1.
> the second is why the delay of producer and consumer is so big when I set
> acks to -1，I think this delay is can not be accepted.
> and I found this delay is amplified with sending more messages.
>
> any feedback is appreciated.
> thanks
>
>
>
>
Re: delay of producer and consumer in kafka 0.9 is too big to be accepted
Posted by Jay Kreps <ja...@confluent.io>.
Can you sanity check this with the end-to-end latency test that ships with
Kafka in the tools package?
https://apache.googlesource.com/kafka/+/1769642bb779921267bd57d3d338591dbdf33842/core/src/main/scala/kafka/tools/TestEndToEndLatency.scala
On Saturday, June 25, 2016, Kafka <ka...@126.com> wrote:
> Hi all，
> my kafka cluster is composed of three brokers with each have 8core
> cpu and 8g memory and 1g network card.
> with java async client，I sent 1000000 messages with size of 1024
> bytes per message ,the send gap between each sending is 20us，the consumer’s
> config is like this，fetch.min.bytes is set to 1, fetch.wait.max.ms is set
> to 100.
> to avoid the inconformity bewteen two machines，I start producer
> and consumer at the same machine，the machine’s configurations have enough
> resources to satisfy these two clients.
>
> I start consumer before producer on each test，with the sending
> timestamp in each message，when consumer receive the message，then I can got
> the consumer delay through the substraction between current timesstamp and
> sending timestamp.
> when I set acks to 0，replica to 2，then the average producer delay
> is 2.98ms， the average consumer delay is 52.23ms.
> when I set acks to 1，replica to 2，then the average producer delay
> is 3.9ms，the average consumer delay is 44.88ms.
> when I set acks to -1, replica to 2, then the average producer
> delay is 1782ms, the average consumer delay is 1786ms.
>
> I have two doubts，the first is why my consumer's delay with acks
> settled to 0 is logger than the consumer delay witch acks settled to 1.
> the second is why the delay of producer and consumer is so big when I set
> acks to -1，I think this delay is can not be accepted.
> and I found this delay is amplified with sending more messages.
>
> any feedback is appreciated.
> thanks
>
>
>
>