You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@kafka.apache.org by Chao Wang <ch...@wustl.edu> on 2017/08/03 19:32:31 UTC

Shooting for microsecond latency between a Kafka producer and a Kafka consumer

Hi,

I observed that it took 2-6 milliseconds for a topic to be received by a 
Kafka consumer from a Kafka producer, and I wonder what I might be 
missing or I was wrong in configuring Kafka for low latency (targeting 
at < 100 microseconds). I did the following:

1. On the broker, I tried to prevent frequent flush of data to disk 
(log.flush.interval.messages=100000)

2. On the producer, I tried to reduce the delay by setting batch.size=0, 
linger.ms=0, acks =0, and I invoked flush() right after send()

3. On the consumer, I set poll(0) (i.e., fetch every data once its 
available?)

I got similar observation (millisecond latency) in varying value size 
from 1 to 512B, and also similar results when either colocating 
producer/consumer or putting them on separate PCs (connecting by a 
switch). As a verification, I implemented simple C/C++ sockets for 
transmission and observed latencies no more than 100 microseconds.

Thanks,

Chao

Re: Shooting for microsecond latency between a Kafka producer and a Kafka consumer

Posted by Chao Wang <ch...@wustl.edu>.

Hi Viktor,

Thanks! I didn't know that. In my case, I was just sending topics 
carrying byte arrays of size 1--512B.

Chao

On 08/11/2017 10:03 AM, Viktor Somogyi wrote:
> Hi Chao,
>
> Apart from the actual request, the consumer sends metadata requests as
> well. What is actually going through the network when you get these
> latencies?
> Try increasing the metadata max age with metadata.max.age.ms.
>
> (Disclaimer: I myself never ran any benchmarking, just trying to help :). )
>
> Viktor
>
> On Mon, Aug 7, 2017 at 7:37 PM, Chao Wang <ch...@wustl.edu> wrote:
>
>> Thanks, David. I was trying to do Kafka pub/sub on a local, closed
>> network. In my case, I observed microsecond-latency with bare-bone sockets,
>> and I would like to know how to configure Kafka to achieve similar result;
>> if it turned out to be infeasible, what might be the cause of the
>> additional latency?
>>
>> Thanks,
>>
>> Chao
>>
>>
>>
>> On 08/07/2017 12:10 PM, David Garcia wrote:
>>
>>> You are not going to get that kind of latency (i.e. less than 100
>>> microseconds).  In my experience, consumer->producer latency averages
>>> around: 20 milliseconds (cluster is in AWS with enhanced networking).
>>>
>>> On 8/3/17, 2:32 PM, "Chao Wang" <ch...@wustl.edu> wrote:
>>>
>>>       Hi,
>>>            I observed that it took 2-6 milliseconds for a topic to be
>>> received by a
>>>       Kafka consumer from a Kafka producer, and I wonder what I might be
>>>       missing or I was wrong in configuring Kafka for low latency
>>> (targeting
>>>       at < 100 microseconds). I did the following:
>>>            1. On the broker, I tried to prevent frequent flush of data to
>>> disk
>>>       (log.flush.interval.messages=100000)
>>>            2. On the producer, I tried to reduce the delay by setting
>>> batch.size=0,
>>>       linger.ms=0, acks =0, and I invoked flush() right after send()
>>>            3. On the consumer, I set poll(0) (i.e., fetch every data once
>>> its
>>>       available?)
>>>            I got similar observation (millisecond latency) in varying
>>> value size
>>>       from 1 to 512B, and also similar results when either colocating
>>>       producer/consumer or putting them on separate PCs (connecting by a
>>>       switch). As a verification, I implemented simple C/C++ sockets for
>>>       transmission and observed latencies no more than 100 microseconds.
>>>            Thanks,
>>>            Chao
>>>
>>>
>>

Re: Shooting for microsecond latency between a Kafka producer and a Kafka consumer

Posted by Viktor Somogyi <vi...@gmail.com>.

Hi Chao,

Apart from the actual request, the consumer sends metadata requests as
well. What is actually going through the network when you get these
latencies?
Try increasing the metadata max age with metadata.max.age.ms.

(Disclaimer: I myself never ran any benchmarking, just trying to help :). )

Viktor

On Mon, Aug 7, 2017 at 7:37 PM, Chao Wang <ch...@wustl.edu> wrote:

> Thanks, David. I was trying to do Kafka pub/sub on a local, closed
> network. In my case, I observed microsecond-latency with bare-bone sockets,
> and I would like to know how to configure Kafka to achieve similar result;
> if it turned out to be infeasible, what might be the cause of the
> additional latency?
>
> Thanks,
>
> Chao
>
>
>
> On 08/07/2017 12:10 PM, David Garcia wrote:
>
>> You are not going to get that kind of latency (i.e. less than 100
>> microseconds).  In my experience, consumer->producer latency averages
>> around: 20 milliseconds (cluster is in AWS with enhanced networking).
>>
>> On 8/3/17, 2:32 PM, "Chao Wang" <ch...@wustl.edu> wrote:
>>
>>      Hi,
>>           I observed that it took 2-6 milliseconds for a topic to be
>> received by a
>>      Kafka consumer from a Kafka producer, and I wonder what I might be
>>      missing or I was wrong in configuring Kafka for low latency
>> (targeting
>>      at < 100 microseconds). I did the following:
>>           1. On the broker, I tried to prevent frequent flush of data to
>> disk
>>      (log.flush.interval.messages=100000)
>>           2. On the producer, I tried to reduce the delay by setting
>> batch.size=0,
>>      linger.ms=0, acks =0, and I invoked flush() right after send()
>>           3. On the consumer, I set poll(0) (i.e., fetch every data once
>> its
>>      available?)
>>           I got similar observation (millisecond latency) in varying
>> value size
>>      from 1 to 512B, and also similar results when either colocating
>>      producer/consumer or putting them on separate PCs (connecting by a
>>      switch). As a verification, I implemented simple C/C++ sockets for
>>      transmission and observed latencies no more than 100 microseconds.
>>           Thanks,
>>           Chao
>>
>>
>
>

Re: Shooting for microsecond latency between a Kafka producer and a Kafka consumer

Posted by Chao Wang <ch...@wustl.edu>.

Thanks, David. I was trying to do Kafka pub/sub on a local, closed 
network. In my case, I observed microsecond-latency with bare-bone 
sockets, and I would like to know how to configure Kafka to achieve 
similar result; if it turned out to be infeasible, what might be the 
cause of the additional latency?

Thanks,

Chao


On 08/07/2017 12:10 PM, David Garcia wrote:
> You are not going to get that kind of latency (i.e. less than 100 microseconds).  In my experience, consumer->producer latency averages around: 20 milliseconds (cluster is in AWS with enhanced networking).
>
> On 8/3/17, 2:32 PM, "Chao Wang" <ch...@wustl.edu> wrote:
>
>      Hi,
>      
>      I observed that it took 2-6 milliseconds for a topic to be received by a
>      Kafka consumer from a Kafka producer, and I wonder what I might be
>      missing or I was wrong in configuring Kafka for low latency (targeting
>      at < 100 microseconds). I did the following:
>      
>      1. On the broker, I tried to prevent frequent flush of data to disk
>      (log.flush.interval.messages=100000)
>      
>      2. On the producer, I tried to reduce the delay by setting batch.size=0,
>      linger.ms=0, acks =0, and I invoked flush() right after send()
>      
>      3. On the consumer, I set poll(0) (i.e., fetch every data once its
>      available?)
>      
>      I got similar observation (millisecond latency) in varying value size
>      from 1 to 512B, and also similar results when either colocating
>      producer/consumer or putting them on separate PCs (connecting by a
>      switch). As a verification, I implemented simple C/C++ sockets for
>      transmission and observed latencies no more than 100 microseconds.
>      
>      Thanks,
>      
>      Chao
>      
>      
>

Re: Shooting for microsecond latency between a Kafka producer and a Kafka consumer

Posted by David Garcia <da...@spiceworks.com>.

You are not going to get that kind of latency (i.e. less than 100 microseconds).  In my experience, consumer->producer latency averages around: 20 milliseconds (cluster is in AWS with enhanced networking).

On 8/3/17, 2:32 PM, "Chao Wang" <ch...@wustl.edu> wrote:

    Hi,
    
    I observed that it took 2-6 milliseconds for a topic to be received by a 
    Kafka consumer from a Kafka producer, and I wonder what I might be 
    missing or I was wrong in configuring Kafka for low latency (targeting 
    at < 100 microseconds). I did the following:
    
    1. On the broker, I tried to prevent frequent flush of data to disk 
    (log.flush.interval.messages=100000)
    
    2. On the producer, I tried to reduce the delay by setting batch.size=0, 
    linger.ms=0, acks =0, and I invoked flush() right after send()
    
    3. On the consumer, I set poll(0) (i.e., fetch every data once its 
    available?)
    
    I got similar observation (millisecond latency) in varying value size 
    from 1 to 512B, and also similar results when either colocating 
    producer/consumer or putting them on separate PCs (connecting by a 
    switch). As a verification, I implemented simple C/C++ sockets for 
    transmission and observed latencies no more than 100 microseconds.
    
    Thanks,
    
    Chao