You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Yuheng Du <yu...@gmail.com> on 2015/07/15 20:36:53 UTC
kafka TestEndtoEndLatency
In kafka performance tests https://gist.github.com/jkreps
/c7ddb4041ef62a900e6c
The TestEndtoEndLatency results are typically around 2ms, while the
ProducerPerformance normally has "average latency"around several hundres ms
when using batch size 8196.
Are both results talking about end to end latency? and the
ProducerPerformance results' latency is just larger because of batching of
several messages?
What is the message size of the TestEndtoEndLatency test uses?
Thanks for any ideas.
best,
Re: kafka TestEndtoEndLatency
Posted by Guozhang Wang <wa...@gmail.com>.
In this case producer.send() should not be blocking since it does not check
on the returned future (i.e. not waiting on the ack).
On Wed, Jul 15, 2015 at 12:57 PM, Yuheng Du <yu...@gmail.com>
wrote:
> Thanks. Here is the source code snippet of EndtoEndLatency test:
>
>
> for (i<- 0 until numMessages) { val begin = System.nanoTime producer.send(
> new ProducerRecord[Array[Byte],Array[Byte]](topic, message)) val received =
> iter.next val elapsed = System.nanoTime - begin // poor man's progress bar
> if (i % 1000 == 0) println(i + "\t" + elapsed / 1000.0 / 1000.0) totalTime
> +
> = elapsed latencies(i) = (elapsed / 1000 / 1000) }
>
>
>
>
>
>
>
>
>
>
>
>
> iter.next reads a message using a consumer process. The
> producer.send()function, is it blocking? I think the message should still
> matters here, am I right?
> In the source code, it is "val message = "hello there beautiful".getBytes"
> is around 21 bytes.
>
> I see that in this EndtoEndLatency test, no acks is involved, right?
>
> Thanks.
> Yuheng
>
> On Wed, Jul 15, 2015 at 3:43 PM, Guozhang Wang <wa...@gmail.com> wrote:
>
> > The end-to-end latency record the transferring of a message from producer
> > to broker, then to consumer.
> >
> > I cannot remember the details not but I think the EndtoEndLatency test
> > record the latency as average, hence it is small.
> >
> > Guozhang
> >
> > On Wed, Jul 15, 2015 at 12:28 PM, Yuheng Du <yu...@gmail.com>
> > wrote:
> >
> > > Guozhang,
> > >
> > > Thank you for explaining. I see that in ProducerPerformance call back
> > > functions were used to get the latency metrics.
> > > For the TestEndtoEndLatency, does message size matter? What this
> > end-to-end
> > > latency comprise of, besides transferring a package from source to
> > > destination (typically around 0.2 ms for 100bytes message)?
> > >
> > > Thanks.
> > >
> > > Yuheng
> > >
> > > On Wed, Jul 15, 2015 at 3:01 PM, Guozhang Wang <wa...@gmail.com>
> > wrote:
> > >
> > > > Yuheng,
> > > >
> > > > Only TestEndtoEndLatency's number are end to end, for
> > ProducerPerformance
> > > > the latency is for the send-to-ack latency, which increases as batch
> > size
> > > > increases.
> > > >
> > > > Guozhang
> > > >
> > > > On Wed, Jul 15, 2015 at 11:36 AM, Yuheng Du <
> yuheng.du.hust@gmail.com>
> > > > wrote:
> > > >
> > > > > In kafka performance tests https://gist.github.com/jkreps
> > > > > /c7ddb4041ef62a900e6c
> > > > >
> > > > > The TestEndtoEndLatency results are typically around 2ms, while the
> > > > > ProducerPerformance normally has "average latency"around several
> > > hundres
> > > > ms
> > > > > when using batch size 8196.
> > > > >
> > > > > Are both results talking about end to end latency? and the
> > > > > ProducerPerformance results' latency is just larger because of
> > batching
> > > > of
> > > > > several messages?
> > > > >
> > > > > What is the message size of the TestEndtoEndLatency test uses?
> > > > >
> > > > > Thanks for any ideas.
> > > > >
> > > > > best,
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > -- Guozhang
> > > >
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>
--
-- Guozhang
Re: kafka TestEndtoEndLatency
Posted by Yuheng Du <yu...@gmail.com>.
Thanks. Here is the source code snippet of EndtoEndLatency test:
for (i<- 0 until numMessages) { val begin = System.nanoTime producer.send(
new ProducerRecord[Array[Byte],Array[Byte]](topic, message)) val received =
iter.next val elapsed = System.nanoTime - begin // poor man's progress bar
if (i % 1000 == 0) println(i + "\t" + elapsed / 1000.0 / 1000.0) totalTime +
= elapsed latencies(i) = (elapsed / 1000 / 1000) }
iter.next reads a message using a consumer process. The
producer.send()function, is it blocking? I think the message should still
matters here, am I right?
In the source code, it is "val message = "hello there beautiful".getBytes"
is around 21 bytes.
I see that in this EndtoEndLatency test, no acks is involved, right?
Thanks.
Yuheng
On Wed, Jul 15, 2015 at 3:43 PM, Guozhang Wang <wa...@gmail.com> wrote:
> The end-to-end latency record the transferring of a message from producer
> to broker, then to consumer.
>
> I cannot remember the details not but I think the EndtoEndLatency test
> record the latency as average, hence it is small.
>
> Guozhang
>
> On Wed, Jul 15, 2015 at 12:28 PM, Yuheng Du <yu...@gmail.com>
> wrote:
>
> > Guozhang,
> >
> > Thank you for explaining. I see that in ProducerPerformance call back
> > functions were used to get the latency metrics.
> > For the TestEndtoEndLatency, does message size matter? What this
> end-to-end
> > latency comprise of, besides transferring a package from source to
> > destination (typically around 0.2 ms for 100bytes message)?
> >
> > Thanks.
> >
> > Yuheng
> >
> > On Wed, Jul 15, 2015 at 3:01 PM, Guozhang Wang <wa...@gmail.com>
> wrote:
> >
> > > Yuheng,
> > >
> > > Only TestEndtoEndLatency's number are end to end, for
> ProducerPerformance
> > > the latency is for the send-to-ack latency, which increases as batch
> size
> > > increases.
> > >
> > > Guozhang
> > >
> > > On Wed, Jul 15, 2015 at 11:36 AM, Yuheng Du <yu...@gmail.com>
> > > wrote:
> > >
> > > > In kafka performance tests https://gist.github.com/jkreps
> > > > /c7ddb4041ef62a900e6c
> > > >
> > > > The TestEndtoEndLatency results are typically around 2ms, while the
> > > > ProducerPerformance normally has "average latency"around several
> > hundres
> > > ms
> > > > when using batch size 8196.
> > > >
> > > > Are both results talking about end to end latency? and the
> > > > ProducerPerformance results' latency is just larger because of
> batching
> > > of
> > > > several messages?
> > > >
> > > > What is the message size of the TestEndtoEndLatency test uses?
> > > >
> > > > Thanks for any ideas.
> > > >
> > > > best,
> > > >
> > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
>
>
>
> --
> -- Guozhang
>
Re: kafka TestEndtoEndLatency
Posted by Guozhang Wang <wa...@gmail.com>.
The end-to-end latency record the transferring of a message from producer
to broker, then to consumer.
I cannot remember the details not but I think the EndtoEndLatency test
record the latency as average, hence it is small.
Guozhang
On Wed, Jul 15, 2015 at 12:28 PM, Yuheng Du <yu...@gmail.com>
wrote:
> Guozhang,
>
> Thank you for explaining. I see that in ProducerPerformance call back
> functions were used to get the latency metrics.
> For the TestEndtoEndLatency, does message size matter? What this end-to-end
> latency comprise of, besides transferring a package from source to
> destination (typically around 0.2 ms for 100bytes message)?
>
> Thanks.
>
> Yuheng
>
> On Wed, Jul 15, 2015 at 3:01 PM, Guozhang Wang <wa...@gmail.com> wrote:
>
> > Yuheng,
> >
> > Only TestEndtoEndLatency's number are end to end, for ProducerPerformance
> > the latency is for the send-to-ack latency, which increases as batch size
> > increases.
> >
> > Guozhang
> >
> > On Wed, Jul 15, 2015 at 11:36 AM, Yuheng Du <yu...@gmail.com>
> > wrote:
> >
> > > In kafka performance tests https://gist.github.com/jkreps
> > > /c7ddb4041ef62a900e6c
> > >
> > > The TestEndtoEndLatency results are typically around 2ms, while the
> > > ProducerPerformance normally has "average latency"around several
> hundres
> > ms
> > > when using batch size 8196.
> > >
> > > Are both results talking about end to end latency? and the
> > > ProducerPerformance results' latency is just larger because of batching
> > of
> > > several messages?
> > >
> > > What is the message size of the TestEndtoEndLatency test uses?
> > >
> > > Thanks for any ideas.
> > >
> > > best,
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>
--
-- Guozhang
Re: kafka TestEndtoEndLatency
Posted by Yuheng Du <yu...@gmail.com>.
Guozhang,
Thank you for explaining. I see that in ProducerPerformance call back
functions were used to get the latency metrics.
For the TestEndtoEndLatency, does message size matter? What this end-to-end
latency comprise of, besides transferring a package from source to
destination (typically around 0.2 ms for 100bytes message)?
Thanks.
Yuheng
On Wed, Jul 15, 2015 at 3:01 PM, Guozhang Wang <wa...@gmail.com> wrote:
> Yuheng,
>
> Only TestEndtoEndLatency's number are end to end, for ProducerPerformance
> the latency is for the send-to-ack latency, which increases as batch size
> increases.
>
> Guozhang
>
> On Wed, Jul 15, 2015 at 11:36 AM, Yuheng Du <yu...@gmail.com>
> wrote:
>
> > In kafka performance tests https://gist.github.com/jkreps
> > /c7ddb4041ef62a900e6c
> >
> > The TestEndtoEndLatency results are typically around 2ms, while the
> > ProducerPerformance normally has "average latency"around several hundres
> ms
> > when using batch size 8196.
> >
> > Are both results talking about end to end latency? and the
> > ProducerPerformance results' latency is just larger because of batching
> of
> > several messages?
> >
> > What is the message size of the TestEndtoEndLatency test uses?
> >
> > Thanks for any ideas.
> >
> > best,
> >
>
>
>
> --
> -- Guozhang
>
Re: kafka TestEndtoEndLatency
Posted by Guozhang Wang <wa...@gmail.com>.
Yuheng,
Only TestEndtoEndLatency's number are end to end, for ProducerPerformance
the latency is for the send-to-ack latency, which increases as batch size
increases.
Guozhang
On Wed, Jul 15, 2015 at 11:36 AM, Yuheng Du <yu...@gmail.com>
wrote:
> In kafka performance tests https://gist.github.com/jkreps
> /c7ddb4041ef62a900e6c
>
> The TestEndtoEndLatency results are typically around 2ms, while the
> ProducerPerformance normally has "average latency"around several hundres ms
> when using batch size 8196.
>
> Are both results talking about end to end latency? and the
> ProducerPerformance results' latency is just larger because of batching of
> several messages?
>
> What is the message size of the TestEndtoEndLatency test uses?
>
> Thanks for any ideas.
>
> best,
>
--
-- Guozhang