You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Yuheng Du <yu...@gmail.com> on 2015/07/14 20:05:19 UTC
Latency test
Currently, the latency test from kafka test the end to end latency between
producers and consumers.
Is there a way to test the producer to broker and broker to consumer
delay seperately?
Thanks.
Re: Latency test
Posted by Yuheng Du <yu...@gmail.com>.
Thank you, Tao!
On Jul 15, 2015 6:27 PM, "Tao Feng" <fe...@gmail.com> wrote:
> Sorry Yufeng, You should change it in $KAFKA_HEAP_OPTS.
>
> On Wed, Jul 15, 2015 at 3:09 PM, Tao Feng <fe...@gmail.com> wrote:
>
> > Hi Yuheng,
> >
> > You could add the -Xmx1024m in
> > https://github.com/apache/kafka/blob/trunk/bin/kafka-run-class.sh
> > KAFKA_JVM_PERFORMANCE_OPTS.
> >
> >
> >
> > On Wed, Jul 15, 2015 at 12:51 AM, Yuheng Du <yu...@gmail.com>
> > wrote:
> >
> >> Tao,
> >>
> >> If I am running on the command line the following command
> >> >bin/kafka-run-class.sh kafka.tools.TestEndToEndLatency
> 192.168.1.3:9092
> >> 192.168.1.1:2181 speedx3 50000000 100 1 -Xmx1024m
> >>
> >> It promped that it is not correct. So where should I put the -Xmx1024m
> >> option? Thanks.
> >>
> >>
> >>
> >> On Wed, Jul 15, 2015 at 3:44 AM, Tao Feng <fe...@gmail.com> wrote:
> >>
> >> > (Please correct me if I am wrong.) Based on TestEndToEndLatency(
> >> >
> >> >
> >>
> https://github.com/apache/kafka/blob/trunk/core/src/test/scala/kafka/tools/TestEndToEndLatency.scala
> >> > ),
> >> > consumer_fetch_max_wait corresponds to fetch.wait.max.ms in consumer
> >> > config(
> >> > http://kafka.apache.org/documentation.html#consumerconfigs). The
> >> default
> >> > value listed at document is 100(ms).
> >> >
> >> > To add java heap space to jvm, put -Xmx$Size(max heap size) for your
> jvm
> >> > option.
> >> >
> >> > On Wed, Jul 15, 2015 at 12:29 AM, Yuheng Du <yuheng.du.hust@gmail.com
> >
> >> > wrote:
> >> >
> >> > > Tao,
> >> > >
> >> > > Thanks. The example on
> >> > https://gist.github.com/jkreps/c7ddb4041ef62a900e6c
> >> > > is outdated already. The error message shows:
> >> > >
> >> > > USAGE: java kafka.tools.TestEndToEndLatency$ broker_list
> >> > zookeeper_connect
> >> > > topic num_messages consumer_fetch_max_wait producer_acks
> >> > >
> >> > >
> >> > > Can anyone helps me what should be put in consumer_fetch_max_wait?
> >> > Thanks.
> >> > >
> >> > > On Tue, Jul 14, 2015 at 5:21 PM, Tao Feng <fe...@gmail.com>
> >> wrote:
> >> > >
> >> > > > I think ProducerPerformance microbenchmark only measure between
> >> client
> >> > > to
> >> > > > brokers(producer to brokers) and provide latency information.
> >> > > >
> >> > > > On Tue, Jul 14, 2015 at 11:05 AM, Yuheng Du <
> >> yuheng.du.hust@gmail.com>
> >> > > > wrote:
> >> > > >
> >> > > > > Currently, the latency test from kafka test the end to end
> latency
> >> > > > between
> >> > > > > producers and consumers.
> >> > > > >
> >> > > > > Is there a way to test the producer to broker and broker to
> >> > consumer
> >> > > > > delay seperately?
> >> > > > >
> >> > > > > Thanks.
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
> >
>
Re: Latency test
Posted by Tao Feng <fe...@gmail.com>.
Sorry Yufeng, You should change it in $KAFKA_HEAP_OPTS.
On Wed, Jul 15, 2015 at 3:09 PM, Tao Feng <fe...@gmail.com> wrote:
> Hi Yuheng,
>
> You could add the -Xmx1024m in
> https://github.com/apache/kafka/blob/trunk/bin/kafka-run-class.sh
> KAFKA_JVM_PERFORMANCE_OPTS.
>
>
>
> On Wed, Jul 15, 2015 at 12:51 AM, Yuheng Du <yu...@gmail.com>
> wrote:
>
>> Tao,
>>
>> If I am running on the command line the following command
>> >bin/kafka-run-class.sh kafka.tools.TestEndToEndLatency 192.168.1.3:9092
>> 192.168.1.1:2181 speedx3 50000000 100 1 -Xmx1024m
>>
>> It promped that it is not correct. So where should I put the -Xmx1024m
>> option? Thanks.
>>
>>
>>
>> On Wed, Jul 15, 2015 at 3:44 AM, Tao Feng <fe...@gmail.com> wrote:
>>
>> > (Please correct me if I am wrong.) Based on TestEndToEndLatency(
>> >
>> >
>> https://github.com/apache/kafka/blob/trunk/core/src/test/scala/kafka/tools/TestEndToEndLatency.scala
>> > ),
>> > consumer_fetch_max_wait corresponds to fetch.wait.max.ms in consumer
>> > config(
>> > http://kafka.apache.org/documentation.html#consumerconfigs). The
>> default
>> > value listed at document is 100(ms).
>> >
>> > To add java heap space to jvm, put -Xmx$Size(max heap size) for your jvm
>> > option.
>> >
>> > On Wed, Jul 15, 2015 at 12:29 AM, Yuheng Du <yu...@gmail.com>
>> > wrote:
>> >
>> > > Tao,
>> > >
>> > > Thanks. The example on
>> > https://gist.github.com/jkreps/c7ddb4041ef62a900e6c
>> > > is outdated already. The error message shows:
>> > >
>> > > USAGE: java kafka.tools.TestEndToEndLatency$ broker_list
>> > zookeeper_connect
>> > > topic num_messages consumer_fetch_max_wait producer_acks
>> > >
>> > >
>> > > Can anyone helps me what should be put in consumer_fetch_max_wait?
>> > Thanks.
>> > >
>> > > On Tue, Jul 14, 2015 at 5:21 PM, Tao Feng <fe...@gmail.com>
>> wrote:
>> > >
>> > > > I think ProducerPerformance microbenchmark only measure between
>> client
>> > > to
>> > > > brokers(producer to brokers) and provide latency information.
>> > > >
>> > > > On Tue, Jul 14, 2015 at 11:05 AM, Yuheng Du <
>> yuheng.du.hust@gmail.com>
>> > > > wrote:
>> > > >
>> > > > > Currently, the latency test from kafka test the end to end latency
>> > > > between
>> > > > > producers and consumers.
>> > > > >
>> > > > > Is there a way to test the producer to broker and broker to
>> > consumer
>> > > > > delay seperately?
>> > > > >
>> > > > > Thanks.
>> > > > >
>> > > >
>> > >
>> >
>>
>
>
Re: Latency test
Posted by Tao Feng <fe...@gmail.com>.
Hi Yuheng,
You could add the -Xmx1024m in
https://github.com/apache/kafka/blob/trunk/bin/kafka-run-class.sh
KAFKA_JVM_PERFORMANCE_OPTS.
On Wed, Jul 15, 2015 at 12:51 AM, Yuheng Du <yu...@gmail.com>
wrote:
> Tao,
>
> If I am running on the command line the following command
> >bin/kafka-run-class.sh kafka.tools.TestEndToEndLatency 192.168.1.3:9092
> 192.168.1.1:2181 speedx3 50000000 100 1 -Xmx1024m
>
> It promped that it is not correct. So where should I put the -Xmx1024m
> option? Thanks.
>
>
>
> On Wed, Jul 15, 2015 at 3:44 AM, Tao Feng <fe...@gmail.com> wrote:
>
> > (Please correct me if I am wrong.) Based on TestEndToEndLatency(
> >
> >
> https://github.com/apache/kafka/blob/trunk/core/src/test/scala/kafka/tools/TestEndToEndLatency.scala
> > ),
> > consumer_fetch_max_wait corresponds to fetch.wait.max.ms in consumer
> > config(
> > http://kafka.apache.org/documentation.html#consumerconfigs). The default
> > value listed at document is 100(ms).
> >
> > To add java heap space to jvm, put -Xmx$Size(max heap size) for your jvm
> > option.
> >
> > On Wed, Jul 15, 2015 at 12:29 AM, Yuheng Du <yu...@gmail.com>
> > wrote:
> >
> > > Tao,
> > >
> > > Thanks. The example on
> > https://gist.github.com/jkreps/c7ddb4041ef62a900e6c
> > > is outdated already. The error message shows:
> > >
> > > USAGE: java kafka.tools.TestEndToEndLatency$ broker_list
> > zookeeper_connect
> > > topic num_messages consumer_fetch_max_wait producer_acks
> > >
> > >
> > > Can anyone helps me what should be put in consumer_fetch_max_wait?
> > Thanks.
> > >
> > > On Tue, Jul 14, 2015 at 5:21 PM, Tao Feng <fe...@gmail.com> wrote:
> > >
> > > > I think ProducerPerformance microbenchmark only measure between
> client
> > > to
> > > > brokers(producer to brokers) and provide latency information.
> > > >
> > > > On Tue, Jul 14, 2015 at 11:05 AM, Yuheng Du <
> yuheng.du.hust@gmail.com>
> > > > wrote:
> > > >
> > > > > Currently, the latency test from kafka test the end to end latency
> > > > between
> > > > > producers and consumers.
> > > > >
> > > > > Is there a way to test the producer to broker and broker to
> > consumer
> > > > > delay seperately?
> > > > >
> > > > > Thanks.
> > > > >
> > > >
> > >
> >
>
Re: Latency test
Posted by Yuheng Du <yu...@gmail.com>.
I have run the end to end latency test and the producerPerformance test on
my kafka cluster according to
https://gist.github.com/jkreps/c7ddb4041ef62a900e6c
In end to end latency test, the latency was around 2ms. In
producerperformance test, if use batch size 8196 to send 50,000,000 records:
>bin/kafka-run-class.sh org.apache.kafka.clients.tools.ProducerPerformance
speedx1 50000000 100 -1 acks=1 bootstrap.servers=192.168.1.1:9092
buffer.memory=67108864 batch.size=8196
The results show that max latency is 3617ms, avg latency 626.7ms. I wanna
know why the latency in producerperformance test is significantly larger
than end to end test? Is it because of batching? Are the definitons of
these two latencies different? I looked at the source code and I believe
the latency is measure for the producer.send() function to complete. So
does this latency includes transmission delay, transferring delay, and what
other components?
Thanks.
best,
Yuheng
On Wed, Jul 15, 2015 at 3:51 AM, Yuheng Du <yu...@gmail.com> wrote:
> Tao,
>
> If I am running on the command line the following command
> >bin/kafka-run-class.sh kafka.tools.TestEndToEndLatency 192.168.1.3:9092
> 192.168.1.1:2181 speedx3 50000000 100 1 -Xmx1024m
>
> It promped that it is not correct. So where should I put the -Xmx1024m
> option? Thanks.
>
>
>
> On Wed, Jul 15, 2015 at 3:44 AM, Tao Feng <fe...@gmail.com> wrote:
>
>> (Please correct me if I am wrong.) Based on TestEndToEndLatency(
>>
>> https://github.com/apache/kafka/blob/trunk/core/src/test/scala/kafka/tools/TestEndToEndLatency.scala
>> ),
>> consumer_fetch_max_wait corresponds to fetch.wait.max.ms in consumer
>> config(
>> http://kafka.apache.org/documentation.html#consumerconfigs). The default
>> value listed at document is 100(ms).
>>
>> To add java heap space to jvm, put -Xmx$Size(max heap size) for your jvm
>> option.
>>
>> On Wed, Jul 15, 2015 at 12:29 AM, Yuheng Du <yu...@gmail.com>
>> wrote:
>>
>> > Tao,
>> >
>> > Thanks. The example on
>> https://gist.github.com/jkreps/c7ddb4041ef62a900e6c
>> > is outdated already. The error message shows:
>> >
>> > USAGE: java kafka.tools.TestEndToEndLatency$ broker_list
>> zookeeper_connect
>> > topic num_messages consumer_fetch_max_wait producer_acks
>> >
>> >
>> > Can anyone helps me what should be put in consumer_fetch_max_wait?
>> Thanks.
>> >
>> > On Tue, Jul 14, 2015 at 5:21 PM, Tao Feng <fe...@gmail.com> wrote:
>> >
>> > > I think ProducerPerformance microbenchmark only measure between
>> client
>> > to
>> > > brokers(producer to brokers) and provide latency information.
>> > >
>> > > On Tue, Jul 14, 2015 at 11:05 AM, Yuheng Du <yuheng.du.hust@gmail.com
>> >
>> > > wrote:
>> > >
>> > > > Currently, the latency test from kafka test the end to end latency
>> > > between
>> > > > producers and consumers.
>> > > >
>> > > > Is there a way to test the producer to broker and broker to
>> consumer
>> > > > delay seperately?
>> > > >
>> > > > Thanks.
>> > > >
>> > >
>> >
>>
>
>
Re: Latency test
Posted by Yuheng Du <yu...@gmail.com>.
Tao,
If I am running on the command line the following command
>bin/kafka-run-class.sh kafka.tools.TestEndToEndLatency 192.168.1.3:9092
192.168.1.1:2181 speedx3 50000000 100 1 -Xmx1024m
It promped that it is not correct. So where should I put the -Xmx1024m
option? Thanks.
On Wed, Jul 15, 2015 at 3:44 AM, Tao Feng <fe...@gmail.com> wrote:
> (Please correct me if I am wrong.) Based on TestEndToEndLatency(
>
> https://github.com/apache/kafka/blob/trunk/core/src/test/scala/kafka/tools/TestEndToEndLatency.scala
> ),
> consumer_fetch_max_wait corresponds to fetch.wait.max.ms in consumer
> config(
> http://kafka.apache.org/documentation.html#consumerconfigs). The default
> value listed at document is 100(ms).
>
> To add java heap space to jvm, put -Xmx$Size(max heap size) for your jvm
> option.
>
> On Wed, Jul 15, 2015 at 12:29 AM, Yuheng Du <yu...@gmail.com>
> wrote:
>
> > Tao,
> >
> > Thanks. The example on
> https://gist.github.com/jkreps/c7ddb4041ef62a900e6c
> > is outdated already. The error message shows:
> >
> > USAGE: java kafka.tools.TestEndToEndLatency$ broker_list
> zookeeper_connect
> > topic num_messages consumer_fetch_max_wait producer_acks
> >
> >
> > Can anyone helps me what should be put in consumer_fetch_max_wait?
> Thanks.
> >
> > On Tue, Jul 14, 2015 at 5:21 PM, Tao Feng <fe...@gmail.com> wrote:
> >
> > > I think ProducerPerformance microbenchmark only measure between client
> > to
> > > brokers(producer to brokers) and provide latency information.
> > >
> > > On Tue, Jul 14, 2015 at 11:05 AM, Yuheng Du <yu...@gmail.com>
> > > wrote:
> > >
> > > > Currently, the latency test from kafka test the end to end latency
> > > between
> > > > producers and consumers.
> > > >
> > > > Is there a way to test the producer to broker and broker to
> consumer
> > > > delay seperately?
> > > >
> > > > Thanks.
> > > >
> > >
> >
>
Re: Latency test
Posted by Tao Feng <fe...@gmail.com>.
(Please correct me if I am wrong.) Based on TestEndToEndLatency(
https://github.com/apache/kafka/blob/trunk/core/src/test/scala/kafka/tools/TestEndToEndLatency.scala),
consumer_fetch_max_wait corresponds to fetch.wait.max.ms in consumer config(
http://kafka.apache.org/documentation.html#consumerconfigs). The default
value listed at document is 100(ms).
To add java heap space to jvm, put -Xmx$Size(max heap size) for your jvm
option.
On Wed, Jul 15, 2015 at 12:29 AM, Yuheng Du <yu...@gmail.com>
wrote:
> Tao,
>
> Thanks. The example on https://gist.github.com/jkreps/c7ddb4041ef62a900e6c
> is outdated already. The error message shows:
>
> USAGE: java kafka.tools.TestEndToEndLatency$ broker_list zookeeper_connect
> topic num_messages consumer_fetch_max_wait producer_acks
>
>
> Can anyone helps me what should be put in consumer_fetch_max_wait? Thanks.
>
> On Tue, Jul 14, 2015 at 5:21 PM, Tao Feng <fe...@gmail.com> wrote:
>
> > I think ProducerPerformance microbenchmark only measure between client
> to
> > brokers(producer to brokers) and provide latency information.
> >
> > On Tue, Jul 14, 2015 at 11:05 AM, Yuheng Du <yu...@gmail.com>
> > wrote:
> >
> > > Currently, the latency test from kafka test the end to end latency
> > between
> > > producers and consumers.
> > >
> > > Is there a way to test the producer to broker and broker to consumer
> > > delay seperately?
> > >
> > > Thanks.
> > >
> >
>
Re: Latency test
Posted by Yuheng Du <yu...@gmail.com>.
I got java out of heap error when running end to end latency test:
yuhengdu@consumer0:/packages/kafka_2.10-0.8.2.1$ bin/kafka-run-class.sh
kafka.tools.TestEndToEndLatency 192.168.1.3:9092 192.168.1.1:2181 speedx3
50000000 100 1
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at kafka.tools.TestEndToEndLatency$.main(TestEndToEndLatency.scala:69)
at kafka.tools.TestEndToEndLatency.main(TestEndToEndLatency.scala)
What command should I do to add java heap space to jvm? Thanks!
Yuheng
On Wed, Jul 15, 2015 at 3:29 AM, Yuheng Du <yu...@gmail.com> wrote:
> Tao,
>
> Thanks. The example on https://gist.github.com/jkreps/c7ddb4041ef62a900e6c
> is outdated already. The error message shows:
>
> USAGE: java kafka.tools.TestEndToEndLatency$ broker_list zookeeper_connect
> topic num_messages consumer_fetch_max_wait producer_acks
>
>
> Can anyone helps me what should be put in consumer_fetch_max_wait? Thanks.
>
> On Tue, Jul 14, 2015 at 5:21 PM, Tao Feng <fe...@gmail.com> wrote:
>
>> I think ProducerPerformance microbenchmark only measure between client to
>> brokers(producer to brokers) and provide latency information.
>>
>> On Tue, Jul 14, 2015 at 11:05 AM, Yuheng Du <yu...@gmail.com>
>> wrote:
>>
>> > Currently, the latency test from kafka test the end to end latency
>> between
>> > producers and consumers.
>> >
>> > Is there a way to test the producer to broker and broker to consumer
>> > delay seperately?
>> >
>> > Thanks.
>> >
>>
>
>
Re: Latency test
Posted by Yuheng Du <yu...@gmail.com>.
Tao,
Thanks. The example on https://gist.github.com/jkreps/c7ddb4041ef62a900e6c
is outdated already. The error message shows:
USAGE: java kafka.tools.TestEndToEndLatency$ broker_list zookeeper_connect
topic num_messages consumer_fetch_max_wait producer_acks
Can anyone helps me what should be put in consumer_fetch_max_wait? Thanks.
On Tue, Jul 14, 2015 at 5:21 PM, Tao Feng <fe...@gmail.com> wrote:
> I think ProducerPerformance microbenchmark only measure between client to
> brokers(producer to brokers) and provide latency information.
>
> On Tue, Jul 14, 2015 at 11:05 AM, Yuheng Du <yu...@gmail.com>
> wrote:
>
> > Currently, the latency test from kafka test the end to end latency
> between
> > producers and consumers.
> >
> > Is there a way to test the producer to broker and broker to consumer
> > delay seperately?
> >
> > Thanks.
> >
>
Re: Latency test
Posted by Tao Feng <fe...@gmail.com>.
I think ProducerPerformance microbenchmark only measure between client to
brokers(producer to brokers) and provide latency information.
On Tue, Jul 14, 2015 at 11:05 AM, Yuheng Du <yu...@gmail.com>
wrote:
> Currently, the latency test from kafka test the end to end latency between
> producers and consumers.
>
> Is there a way to test the producer to broker and broker to consumer
> delay seperately?
>
> Thanks.
>