You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Supun Kamburugamuva <su...@gmail.com> on 2014/06/18 16:31:19 UTC
Kafka latency measures
Hi,
We are trying to evaluate Kafka for a real time application. We are sending
50 Kb messages at a fixed rate. The normal messages have a reasonable
latency. But then there are these outliers that takes unpredictable amount
of time. This causes the average latency to increase dramatically. We are
running with basically the default configuration. Any suggestions for
improving the latency?
Thanks in advance,
Supun..
--
Supun Kamburugamuva
Member, Apache Software Foundation; http://www.apache.org
E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
Blog: http://supunk.blogspot.com
Re: Kafka latency measures
Posted by Jay Kreps <ja...@gmail.com>.
There were actually several patches against trunk since 0.8.1.1 that may
impact latency however, especially when using acks=-1. So those results in
the blog may be a bit better than what you would see in 0.8.1.1.
-Jay
On Wed, Jun 18, 2014 at 7:58 PM, Supun Kamburugamuva <su...@gmail.com>
wrote:
> My machine configuration is not very high. The average one way latency we
> observe is around 10 ~ 15 ms for 50k messages. The outliers doesn't occur
> for small messages. For small messages we observe around 6 ms latency.
>
> Thanks,
> Supun..
>
>
> On Wed, Jun 18, 2014 at 10:18 PM, Neha Narkhede <ne...@gmail.com>
> wrote:
>
> > what are the latency numbers you observed, avg as well as worst case?
> Here
> > is a blog that we did recently which should reflect latest performance
> > metrics for latency -
> >
> >
> http://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines
> >
> >
> > On Wed, Jun 18, 2014 at 11:01 AM, Supun Kamburugamuva <supun06@gmail.com
> >
> > wrote:
> >
> > > I've found this performance test.
> > >
> > > http://blog.liveramp.com/2013/04/08/kafka-0-8-producer-performance-2/
> > >
> > > This performance test has mentioned about the same issue at the end.
> > >
> > > Thanks,
> > > Supun..
> > >
> > >
> > > On Wed, Jun 18, 2014 at 12:43 PM, Supun Kamburugamuva <
> supun06@gmail.com
> > >
> > > wrote:
> > >
> > > > The spikes happens without any correlation with the
> > > log.flush.interval.message.
> > > > They happen more frequently.
> > > >
> > > > I'm using the latest version. I'm sending the messages to Kafka, then
> > > > there is a message receiver, it sends the same messages back through
> > > kafka
> > > > to original sender. The round trip latency is measured.
> > > >
> > > > Thanks,
> > > > Supun..
> > > >
> > > >
> > > > On Wed, Jun 18, 2014 at 12:02 PM, Neha Narkhede <
> > neha.narkhede@gmail.com
> > > >
> > > > wrote:
> > > >
> > > >> Which version of Kafka did you use? When you say latency, do you
> mean
> > > the
> > > >> latency between the producer and consumer? If so, are you using a
> > > >> timestamp
> > > >> within the message to compute this latency?
> > > >>
> > > >>
> > > >> On Wed, Jun 18, 2014 at 8:15 AM, Magnus Edenhill <
> magnus@edenhill.se>
> > > >> wrote:
> > > >>
> > > >> > Hi,
> > > >> >
> > > >> > do these spikes happen to correlate with
> log.flush.interval.messages
> > > or
> > > >> > log.flush.interval.ms?
> > > >> > If so it's the file system sync blockage you are seeing.
> > > >> >
> > > >> > /Magnus
> > > >> >
> > > >> >
> > > >> > 2014-06-18 16:31 GMT+02:00 Supun Kamburugamuva <supun06@gmail.com
> >:
> > > >> >
> > > >> > > Hi,
> > > >> > >
> > > >> > > We are trying to evaluate Kafka for a real time application. We
> > are
> > > >> > sending
> > > >> > > 50 Kb messages at a fixed rate. The normal messages have a
> > > reasonable
> > > >> > > latency. But then there are these outliers that takes
> > unpredictable
> > > >> > amount
> > > >> > > of time. This causes the average latency to increase
> dramatically.
> > > We
> > > >> are
> > > >> > > running with basically the default configuration. Any
> suggestions
> > > for
> > > >> > > improving the latency?
> > > >> > >
> > > >> > > Thanks in advance,
> > > >> > > Supun..
> > > >> > >
> > > >> > > --
> > > >> > > Supun Kamburugamuva
> > > >> > > Member, Apache Software Foundation; http://www.apache.org
> > > >> > > E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
> > > >> > > Blog: http://supunk.blogspot.com
> > > >> > >
> > > >> >
> > > >>
> > > >
> > > >
> > > >
> > > > --
> > > > Supun Kamburugamuva
> > > > Member, Apache Software Foundation; http://www.apache.org
> > > > E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
> > > > Blog: http://supunk.blogspot.com
> > > >
> > > >
> > >
> > >
> > > --
> > > Supun Kamburugamuva
> > > Member, Apache Software Foundation; http://www.apache.org
> > > E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
> > > Blog: http://supunk.blogspot.com
> > >
> >
>
>
>
> --
> Supun Kamburugamuva
> Member, Apache Software Foundation; http://www.apache.org
> E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
> Blog: http://supunk.blogspot.com
>
Re: Kafka latency measures
Posted by Supun Kamburugamuva <su...@gmail.com>.
My machine configuration is not very high. The average one way latency we
observe is around 10 ~ 15 ms for 50k messages. The outliers doesn't occur
for small messages. For small messages we observe around 6 ms latency.
Thanks,
Supun..
On Wed, Jun 18, 2014 at 10:18 PM, Neha Narkhede <ne...@gmail.com>
wrote:
> what are the latency numbers you observed, avg as well as worst case? Here
> is a blog that we did recently which should reflect latest performance
> metrics for latency -
>
> http://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines
>
>
> On Wed, Jun 18, 2014 at 11:01 AM, Supun Kamburugamuva <su...@gmail.com>
> wrote:
>
> > I've found this performance test.
> >
> > http://blog.liveramp.com/2013/04/08/kafka-0-8-producer-performance-2/
> >
> > This performance test has mentioned about the same issue at the end.
> >
> > Thanks,
> > Supun..
> >
> >
> > On Wed, Jun 18, 2014 at 12:43 PM, Supun Kamburugamuva <supun06@gmail.com
> >
> > wrote:
> >
> > > The spikes happens without any correlation with the
> > log.flush.interval.message.
> > > They happen more frequently.
> > >
> > > I'm using the latest version. I'm sending the messages to Kafka, then
> > > there is a message receiver, it sends the same messages back through
> > kafka
> > > to original sender. The round trip latency is measured.
> > >
> > > Thanks,
> > > Supun..
> > >
> > >
> > > On Wed, Jun 18, 2014 at 12:02 PM, Neha Narkhede <
> neha.narkhede@gmail.com
> > >
> > > wrote:
> > >
> > >> Which version of Kafka did you use? When you say latency, do you mean
> > the
> > >> latency between the producer and consumer? If so, are you using a
> > >> timestamp
> > >> within the message to compute this latency?
> > >>
> > >>
> > >> On Wed, Jun 18, 2014 at 8:15 AM, Magnus Edenhill <ma...@edenhill.se>
> > >> wrote:
> > >>
> > >> > Hi,
> > >> >
> > >> > do these spikes happen to correlate with log.flush.interval.messages
> > or
> > >> > log.flush.interval.ms?
> > >> > If so it's the file system sync blockage you are seeing.
> > >> >
> > >> > /Magnus
> > >> >
> > >> >
> > >> > 2014-06-18 16:31 GMT+02:00 Supun Kamburugamuva <su...@gmail.com>:
> > >> >
> > >> > > Hi,
> > >> > >
> > >> > > We are trying to evaluate Kafka for a real time application. We
> are
> > >> > sending
> > >> > > 50 Kb messages at a fixed rate. The normal messages have a
> > reasonable
> > >> > > latency. But then there are these outliers that takes
> unpredictable
> > >> > amount
> > >> > > of time. This causes the average latency to increase dramatically.
> > We
> > >> are
> > >> > > running with basically the default configuration. Any suggestions
> > for
> > >> > > improving the latency?
> > >> > >
> > >> > > Thanks in advance,
> > >> > > Supun..
> > >> > >
> > >> > > --
> > >> > > Supun Kamburugamuva
> > >> > > Member, Apache Software Foundation; http://www.apache.org
> > >> > > E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
> > >> > > Blog: http://supunk.blogspot.com
> > >> > >
> > >> >
> > >>
> > >
> > >
> > >
> > > --
> > > Supun Kamburugamuva
> > > Member, Apache Software Foundation; http://www.apache.org
> > > E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
> > > Blog: http://supunk.blogspot.com
> > >
> > >
> >
> >
> > --
> > Supun Kamburugamuva
> > Member, Apache Software Foundation; http://www.apache.org
> > E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
> > Blog: http://supunk.blogspot.com
> >
>
--
Supun Kamburugamuva
Member, Apache Software Foundation; http://www.apache.org
E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
Blog: http://supunk.blogspot.com
Re: Kafka latency measures
Posted by Neha Narkhede <ne...@gmail.com>.
what are the latency numbers you observed, avg as well as worst case? Here
is a blog that we did recently which should reflect latest performance
metrics for latency -
http://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines
On Wed, Jun 18, 2014 at 11:01 AM, Supun Kamburugamuva <su...@gmail.com>
wrote:
> I've found this performance test.
>
> http://blog.liveramp.com/2013/04/08/kafka-0-8-producer-performance-2/
>
> This performance test has mentioned about the same issue at the end.
>
> Thanks,
> Supun..
>
>
> On Wed, Jun 18, 2014 at 12:43 PM, Supun Kamburugamuva <su...@gmail.com>
> wrote:
>
> > The spikes happens without any correlation with the
> log.flush.interval.message.
> > They happen more frequently.
> >
> > I'm using the latest version. I'm sending the messages to Kafka, then
> > there is a message receiver, it sends the same messages back through
> kafka
> > to original sender. The round trip latency is measured.
> >
> > Thanks,
> > Supun..
> >
> >
> > On Wed, Jun 18, 2014 at 12:02 PM, Neha Narkhede <neha.narkhede@gmail.com
> >
> > wrote:
> >
> >> Which version of Kafka did you use? When you say latency, do you mean
> the
> >> latency between the producer and consumer? If so, are you using a
> >> timestamp
> >> within the message to compute this latency?
> >>
> >>
> >> On Wed, Jun 18, 2014 at 8:15 AM, Magnus Edenhill <ma...@edenhill.se>
> >> wrote:
> >>
> >> > Hi,
> >> >
> >> > do these spikes happen to correlate with log.flush.interval.messages
> or
> >> > log.flush.interval.ms?
> >> > If so it's the file system sync blockage you are seeing.
> >> >
> >> > /Magnus
> >> >
> >> >
> >> > 2014-06-18 16:31 GMT+02:00 Supun Kamburugamuva <su...@gmail.com>:
> >> >
> >> > > Hi,
> >> > >
> >> > > We are trying to evaluate Kafka for a real time application. We are
> >> > sending
> >> > > 50 Kb messages at a fixed rate. The normal messages have a
> reasonable
> >> > > latency. But then there are these outliers that takes unpredictable
> >> > amount
> >> > > of time. This causes the average latency to increase dramatically.
> We
> >> are
> >> > > running with basically the default configuration. Any suggestions
> for
> >> > > improving the latency?
> >> > >
> >> > > Thanks in advance,
> >> > > Supun..
> >> > >
> >> > > --
> >> > > Supun Kamburugamuva
> >> > > Member, Apache Software Foundation; http://www.apache.org
> >> > > E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
> >> > > Blog: http://supunk.blogspot.com
> >> > >
> >> >
> >>
> >
> >
> >
> > --
> > Supun Kamburugamuva
> > Member, Apache Software Foundation; http://www.apache.org
> > E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
> > Blog: http://supunk.blogspot.com
> >
> >
>
>
> --
> Supun Kamburugamuva
> Member, Apache Software Foundation; http://www.apache.org
> E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
> Blog: http://supunk.blogspot.com
>
Re: Kafka latency measures
Posted by Supun Kamburugamuva <su...@gmail.com>.
I've found this performance test.
http://blog.liveramp.com/2013/04/08/kafka-0-8-producer-performance-2/
This performance test has mentioned about the same issue at the end.
Thanks,
Supun..
On Wed, Jun 18, 2014 at 12:43 PM, Supun Kamburugamuva <su...@gmail.com>
wrote:
> The spikes happens without any correlation with the log.flush.interval.message.
> They happen more frequently.
>
> I'm using the latest version. I'm sending the messages to Kafka, then
> there is a message receiver, it sends the same messages back through kafka
> to original sender. The round trip latency is measured.
>
> Thanks,
> Supun..
>
>
> On Wed, Jun 18, 2014 at 12:02 PM, Neha Narkhede <ne...@gmail.com>
> wrote:
>
>> Which version of Kafka did you use? When you say latency, do you mean the
>> latency between the producer and consumer? If so, are you using a
>> timestamp
>> within the message to compute this latency?
>>
>>
>> On Wed, Jun 18, 2014 at 8:15 AM, Magnus Edenhill <ma...@edenhill.se>
>> wrote:
>>
>> > Hi,
>> >
>> > do these spikes happen to correlate with log.flush.interval.messages or
>> > log.flush.interval.ms?
>> > If so it's the file system sync blockage you are seeing.
>> >
>> > /Magnus
>> >
>> >
>> > 2014-06-18 16:31 GMT+02:00 Supun Kamburugamuva <su...@gmail.com>:
>> >
>> > > Hi,
>> > >
>> > > We are trying to evaluate Kafka for a real time application. We are
>> > sending
>> > > 50 Kb messages at a fixed rate. The normal messages have a reasonable
>> > > latency. But then there are these outliers that takes unpredictable
>> > amount
>> > > of time. This causes the average latency to increase dramatically. We
>> are
>> > > running with basically the default configuration. Any suggestions for
>> > > improving the latency?
>> > >
>> > > Thanks in advance,
>> > > Supun..
>> > >
>> > > --
>> > > Supun Kamburugamuva
>> > > Member, Apache Software Foundation; http://www.apache.org
>> > > E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
>> > > Blog: http://supunk.blogspot.com
>> > >
>> >
>>
>
>
>
> --
> Supun Kamburugamuva
> Member, Apache Software Foundation; http://www.apache.org
> E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
> Blog: http://supunk.blogspot.com
>
>
--
Supun Kamburugamuva
Member, Apache Software Foundation; http://www.apache.org
E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
Blog: http://supunk.blogspot.com
Re: Kafka latency measures
Posted by Supun Kamburugamuva <su...@gmail.com>.
The spikes happens without any correlation with the
log.flush.interval.message.
They happen more frequently.
I'm using the latest version. I'm sending the messages to Kafka, then there
is a message receiver, it sends the same messages back through kafka to
original sender. The round trip latency is measured.
Thanks,
Supun..
On Wed, Jun 18, 2014 at 12:02 PM, Neha Narkhede <ne...@gmail.com>
wrote:
> Which version of Kafka did you use? When you say latency, do you mean the
> latency between the producer and consumer? If so, are you using a timestamp
> within the message to compute this latency?
>
>
> On Wed, Jun 18, 2014 at 8:15 AM, Magnus Edenhill <ma...@edenhill.se>
> wrote:
>
> > Hi,
> >
> > do these spikes happen to correlate with log.flush.interval.messages or
> > log.flush.interval.ms?
> > If so it's the file system sync blockage you are seeing.
> >
> > /Magnus
> >
> >
> > 2014-06-18 16:31 GMT+02:00 Supun Kamburugamuva <su...@gmail.com>:
> >
> > > Hi,
> > >
> > > We are trying to evaluate Kafka for a real time application. We are
> > sending
> > > 50 Kb messages at a fixed rate. The normal messages have a reasonable
> > > latency. But then there are these outliers that takes unpredictable
> > amount
> > > of time. This causes the average latency to increase dramatically. We
> are
> > > running with basically the default configuration. Any suggestions for
> > > improving the latency?
> > >
> > > Thanks in advance,
> > > Supun..
> > >
> > > --
> > > Supun Kamburugamuva
> > > Member, Apache Software Foundation; http://www.apache.org
> > > E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
> > > Blog: http://supunk.blogspot.com
> > >
> >
>
--
Supun Kamburugamuva
Member, Apache Software Foundation; http://www.apache.org
E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
Blog: http://supunk.blogspot.com
Re: Kafka latency measures
Posted by Neha Narkhede <ne...@gmail.com>.
Which version of Kafka did you use? When you say latency, do you mean the
latency between the producer and consumer? If so, are you using a timestamp
within the message to compute this latency?
On Wed, Jun 18, 2014 at 8:15 AM, Magnus Edenhill <ma...@edenhill.se> wrote:
> Hi,
>
> do these spikes happen to correlate with log.flush.interval.messages or
> log.flush.interval.ms?
> If so it's the file system sync blockage you are seeing.
>
> /Magnus
>
>
> 2014-06-18 16:31 GMT+02:00 Supun Kamburugamuva <su...@gmail.com>:
>
> > Hi,
> >
> > We are trying to evaluate Kafka for a real time application. We are
> sending
> > 50 Kb messages at a fixed rate. The normal messages have a reasonable
> > latency. But then there are these outliers that takes unpredictable
> amount
> > of time. This causes the average latency to increase dramatically. We are
> > running with basically the default configuration. Any suggestions for
> > improving the latency?
> >
> > Thanks in advance,
> > Supun..
> >
> > --
> > Supun Kamburugamuva
> > Member, Apache Software Foundation; http://www.apache.org
> > E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
> > Blog: http://supunk.blogspot.com
> >
>
Re: Kafka latency measures
Posted by Magnus Edenhill <ma...@edenhill.se>.
Hi,
do these spikes happen to correlate with log.flush.interval.messages or
log.flush.interval.ms?
If so it's the file system sync blockage you are seeing.
/Magnus
2014-06-18 16:31 GMT+02:00 Supun Kamburugamuva <su...@gmail.com>:
> Hi,
>
> We are trying to evaluate Kafka for a real time application. We are sending
> 50 Kb messages at a fixed rate. The normal messages have a reasonable
> latency. But then there are these outliers that takes unpredictable amount
> of time. This causes the average latency to increase dramatically. We are
> running with basically the default configuration. Any suggestions for
> improving the latency?
>
> Thanks in advance,
> Supun..
>
> --
> Supun Kamburugamuva
> Member, Apache Software Foundation; http://www.apache.org
> E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
> Blog: http://supunk.blogspot.com
>