Time difference between message fetch and message send is very high (~900-1200ms)

["Kishore V. Kopalle" <ki...@greenmedsoft.com>]

Hello All,

In my earlier mail, I am sorry I did not give lot of details on my setup
where I am measuring performance between producer and  consumer. Here are
the details:

- I am using Kafka version 0.7 downloaded from Apache web site.
- I am using a modified version of KafkaConsumerProducerDemo example that
comes with  the distribution. I have made the following modifications:

- In Producer java file:

    int messageNo = 1;
    SimpleDateFormat dateFormat = new SimpleDateFormat("yyyy-MM-dd
HH:mm:ss.SSS") ;

    while(true)
    {
      String messageStr = new String("Message_" + messageNo);
      producer.send(new ProducerData<Integer, String>(topic,
       "Message produced at " + dateFormat.format(new Date())
       + ": " + messageStr));
      messageNo++;
    }


- In Consumer.java file:

    SimpleDateFormat dateFormat = new SimpleDateFormat("yyyy-MM-dd
HH:mm:ss.SSS") ;

    while(it.hasNext())
      System.out.println("Message received at " + dateFormat.format(new
Date()) + ": " + ExampleUtils.getMessage(it.next().message()));


Based on the messages on the console from consumer, I am getting values
ranging from 900ms to 1200ms per message for the time difference between
received message time and the timestamp in the message itself.

Kindly let me know where I am going wrong and are there any configuration
steps involved for getting optimal performance for the time taken for a
specific message to be fetched and the time when it was sent by producer.

Regards,
Kishore

Re: Time difference between message fetch and message send is very high (~900-1200ms)

[Jun Rao <ju...@gmail.com>]

There are mainly 2 things to consider for latency. (1) How quickly does the
producer send the message to the broker? If you use the sync mode in the
producer, the message will be sent immediately (throughput will be lower in
the sync mode though). If you use the async mode, the messages are sent
after enough messages have accumulated or enough time has passed. To reduce
latency, you need to reduce one or both thresholds. (2) How quickly does a
message get exposed to the consumer after it reaches the broker? In 0.7, a
message is exposed after it's flushed to disk.  By default, we flush the
log every 500 messages or 3 secs, for lower latency, you can try to reduce
log.flush.interval and default.log.flush.interval.ms. In 0.8, a message is
exposed to the consumer as soon as it reaches multiple brokers in memory.
So the latency will be better.

Thanks,

Jun


On Thu, May 16, 2013 at 10:37 PM, Kishore V. Kopalle <
kishore@greenmedsoft.com> wrote:

> Hello All,
>
> In my earlier mail, I am sorry I did not give lot of details on my setup
> where I am measuring performance between producer and  consumer. Here are
> the details:
>
> - I am using Kafka version 0.7 downloaded from Apache web site.
> - I am using a modified version of KafkaConsumerProducerDemo example that
> comes with  the distribution. I have made the following modifications:
>
> - In Producer java file:
>
>     int messageNo = 1;
>     SimpleDateFormat dateFormat = new SimpleDateFormat("yyyy-MM-dd
> HH:mm:ss.SSS") ;
>
>     while(true)
>     {
>       String messageStr = new String("Message_" + messageNo);
>       producer.send(new ProducerData<Integer, String>(topic,
>        "Message produced at " + dateFormat.format(new Date())
>        + ": " + messageStr));
>       messageNo++;
>     }
>
>
> - In Consumer.java file:
>
>     SimpleDateFormat dateFormat = new SimpleDateFormat("yyyy-MM-dd
> HH:mm:ss.SSS") ;
>
>     while(it.hasNext())
>       System.out.println("Message received at " + dateFormat.format(new
> Date()) + ": " + ExampleUtils.getMessage(it.next().message()));
>
>
> Based on the messages on the console from consumer, I am getting values
> ranging from 900ms to 1200ms per message for the time difference between
> received message time and the timestamp in the message itself.
>
> Kindly let me know where I am going wrong and are there any configuration
> steps involved for getting optimal performance for the time taken for a
> specific message to be fetched and the time when it was sent by producer.
>
> Regards,
> Kishore
>