You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Klaus Schaefers <kl...@ligatus.com> on 2014/01/10 10:43:56 UTC

Velocity on local machine

Hi,

I am currently benchmarking Kafka against ActiveMQ and I got some results
the surprised my quite a lot. ActiveMQ managed to deliver 4x more messages
when running locally. But from all what I was reading I am a little bit
surprised. Honestly I expected Kafka to outperform ActiveMQ. Some I am left
with the feeling that I configured some wrong or did any kind of other
mistake.

My setup looks like this:

- One local broker
- 10 Topic / Queues
- 1 producer that dispatches messages randomly to the topics
- 1 consumer per topic

I basically used the example on the Kafka web page.

Also I encountered some issues when increasing the number of topics, lets
say 100. In this case the consumer cannot connect the Zookeeper...

Does anybody has an idea how to improve the performance?


Thx,

Klaus


-- 

-- 

Klaus Schaefers
Senior Optimization Manager

Ligatus GmbH
Hohenstaufenring 30-32
D-50674 Köln

Tel.:  +49 (0) 221 / 56939 -784
Fax:  +49 (0) 221 / 56 939 - 599
E-Mail: klaus.schaefers@ligatus.com
Web: www.ligatus.de

HRB Köln 56003
Geschäftsführung:
Dipl.-Kaufmann Lars Hasselbach, Dipl.-Kaufmann Klaus Ludemann,
Dipl.-Wirtschaftsingenieur Arne Wolter

Re: Velocity on local machine

Posted by Klaus Schaefers <kl...@ligatus.com>.
Hi,

I am using kafka-0.8.0-src.

Cheers,

Klaus


On Fri, Jan 10, 2014 at 8:48 PM, Neha Narkhede <ne...@gmail.com>wrote:

> What version of Kafka are you benchmarking?
>
>
> On Fri, Jan 10, 2014 at 8:36 AM, Klaus Schaefers <
> klaus.schaefers@ligatus.com> wrote:
>
> > Hi,
> >
> > during my test cpu load is quite low, roughly 50 percent, sometimes peask
> > to 70%.
> >
> > >Are you using the sync producer per chance?
> > I enforced now the async and I got a huge improvement in one of my test
> > cases. let my explore the rest a little bit more.
> >
> > Cheers,
> >
> > Klaus
> >
> >
> >
> >
> > On Fri, Jan 10, 2014 at 2:36 PM, Magnus Edenhill <ma...@edenhill.se>
> > wrote:
> >
> > > 2k msgs/s is silly, unless your messages are 10MB each, so something is
> > > clearly wrong.
> > >
> > > * What is the CPU usage and IO load when running your performance
> tests?
> > > * Are you using the sync producer per chance? Maybe combined with an
> > > aggressive log.flush.interval?
> > > * For reference, and to find out where the bottleneck is, try running
> > > rdkafka_performance [1] according to the previous link I posted:
> > >   if performance does not increase then the broker is the problem,
> > > otherwise it is the producer/consumers.
> > >
> > > Regards,
> > > Magnus
> > >
> > > [1]: https://github.com/edenhill/librdkafka/tree/master/examples
> > >
> > >
> > >
> > > 2014/1/10 Klaus Schaefers <kl...@ligatus.com>
> > >
> > > > Hi,
> > > >
> > > > I have close to 2k messages per second. My machine is just a (BG
> 4-core
> > > i5
> > > > but I would expect more messages. I ran Kafka in the default
> settings.
> > > >
> > > >
> > > > On Fri, Jan 10, 2014 at 12:31 PM, Magnus Edenhill <
> magnus@edenhill.se
> > > > >wrote:
> > > >
> > > > > What performance numbers did you see?
> > > > >
> > > > > For reference you can check the following tests that were also run
> on
> > > the
> > > > > same machine as the broker:
> > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/edenhill/librdkafka/blob/master/INTRODUCTION.md#performance-numbers
> > > > >
> > > > > Do they correspond to your numbers?
> > > > >
> > > > > Consumer thruput is not included in that document but typically
> peaks
> > > at
> > > > > around 3 million msgs/s when run on the same host and with hot disk
> > > > caches.
> > > > >
> > > > > Regards,
> > > > > Magnus
> > > > >
> > > > >
> > > > > 2014/1/10 Klaus Schaefers <kl...@ligatus.com>
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I am currently benchmarking Kafka against ActiveMQ and I got some
> > > > results
> > > > > > the surprised my quite a lot. ActiveMQ managed to deliver 4x more
> > > > > messages
> > > > > > when running locally. But from all what I was reading I am a
> little
> > > bit
> > > > > > surprised. Honestly I expected Kafka to outperform ActiveMQ.
> Some I
> > > am
> > > > > left
> > > > > > with the feeling that I configured some wrong or did any kind of
> > > other
> > > > > > mistake.
> > > > > >
> > > > > > My setup looks like this:
> > > > > >
> > > > > > - One local broker
> > > > > > - 10 Topic / Queues
> > > > > > - 1 producer that dispatches messages randomly to the topics
> > > > > > - 1 consumer per topic
> > > > > >
> > > > > > I basically used the example on the Kafka web page.
> > > > > >
> > > > > > Also I encountered some issues when increasing the number of
> > topics,
> > > > lets
> > > > > > say 100. In this case the consumer cannot connect the
> Zookeeper...
> > > > > >
> > > > > > Does anybody has an idea how to improve the performance?
> > > > > >
> > > > > >
> > > > > > Thx,
> > > > > >
> > > > > > Klaus
> > > > > >
> > > > > >
> > > > > > --
> > > > > >
> > > > > > --
> > > > > >
> > > > > > Klaus Schaefers
> > > > > > Senior Optimization Manager
> > > > > >
> > > > > > Ligatus GmbH
> > > > > > Hohenstaufenring 30-32
> > > > > > D-50674 Köln
> > > > > >
> > > > > > Tel.:  +49 (0) 221 / 56939 -784
> > > > > > Fax:  +49 (0) 221 / 56 939 - 599
> > > > > > E-Mail: klaus.schaefers@ligatus.com
> > > > > > Web: www.ligatus.de
> > > > > >
> > > > > > HRB Köln 56003
> > > > > > Geschäftsführung:
> > > > > > Dipl.-Kaufmann Lars Hasselbach, Dipl.-Kaufmann Klaus Ludemann,
> > > > > > Dipl.-Wirtschaftsingenieur Arne Wolter
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > >
> > > > --
> > > >
> > > > Klaus Schaefers
> > > > Senior Optimization Manager
> > > >
> > > > Ligatus GmbH
> > > > Hohenstaufenring 30-32
> > > > D-50674 Köln
> > > >
> > > > Tel.:  +49 (0) 221 / 56939 -784
> > > > Fax:  +49 (0) 221 / 56 939 - 599
> > > > E-Mail: klaus.schaefers@ligatus.com
> > > > Web: www.ligatus.de
> > > >
> > > > HRB Köln 56003
> > > > Geschäftsführung:
> > > > Dipl.-Kaufmann Lars Hasselbach, Dipl.-Kaufmann Klaus Ludemann,
> > > > Dipl.-Wirtschaftsingenieur Arne Wolter
> > > >
> > >
> >
> >
> >
> > --
> >
> > --
> >
> > Klaus Schaefers
> > Senior Optimization Manager
> >
> > Ligatus GmbH
> > Hohenstaufenring 30-32
> > D-50674 Köln
> >
> > Tel.:  +49 (0) 221 / 56939 -784
> > Fax:  +49 (0) 221 / 56 939 - 599
> > E-Mail: klaus.schaefers@ligatus.com
> > Web: www.ligatus.de
> >
> > HRB Köln 56003
> > Geschäftsführung:
> > Dipl.-Kaufmann Lars Hasselbach, Dipl.-Kaufmann Klaus Ludemann,
> > Dipl.-Wirtschaftsingenieur Arne Wolter
> >
>



-- 

-- 

Klaus Schaefers
Senior Optimization Manager

Ligatus GmbH
Hohenstaufenring 30-32
D-50674 Köln

Tel.:  +49 (0) 221 / 56939 -784
Fax:  +49 (0) 221 / 56 939 - 599
E-Mail: klaus.schaefers@ligatus.com
Web: www.ligatus.de

HRB Köln 56003
Geschäftsführung:
Dipl.-Kaufmann Lars Hasselbach, Dipl.-Kaufmann Klaus Ludemann,
Dipl.-Wirtschaftsingenieur Arne Wolter

Re: Velocity on local machine

Posted by Neha Narkhede <ne...@gmail.com>.
What version of Kafka are you benchmarking?


On Fri, Jan 10, 2014 at 8:36 AM, Klaus Schaefers <
klaus.schaefers@ligatus.com> wrote:

> Hi,
>
> during my test cpu load is quite low, roughly 50 percent, sometimes peask
> to 70%.
>
> >Are you using the sync producer per chance?
> I enforced now the async and I got a huge improvement in one of my test
> cases. let my explore the rest a little bit more.
>
> Cheers,
>
> Klaus
>
>
>
>
> On Fri, Jan 10, 2014 at 2:36 PM, Magnus Edenhill <ma...@edenhill.se>
> wrote:
>
> > 2k msgs/s is silly, unless your messages are 10MB each, so something is
> > clearly wrong.
> >
> > * What is the CPU usage and IO load when running your performance tests?
> > * Are you using the sync producer per chance? Maybe combined with an
> > aggressive log.flush.interval?
> > * For reference, and to find out where the bottleneck is, try running
> > rdkafka_performance [1] according to the previous link I posted:
> >   if performance does not increase then the broker is the problem,
> > otherwise it is the producer/consumers.
> >
> > Regards,
> > Magnus
> >
> > [1]: https://github.com/edenhill/librdkafka/tree/master/examples
> >
> >
> >
> > 2014/1/10 Klaus Schaefers <kl...@ligatus.com>
> >
> > > Hi,
> > >
> > > I have close to 2k messages per second. My machine is just a (BG 4-core
> > i5
> > > but I would expect more messages. I ran Kafka in the default settings.
> > >
> > >
> > > On Fri, Jan 10, 2014 at 12:31 PM, Magnus Edenhill <magnus@edenhill.se
> > > >wrote:
> > >
> > > > What performance numbers did you see?
> > > >
> > > > For reference you can check the following tests that were also run on
> > the
> > > > same machine as the broker:
> > > >
> > > >
> > >
> >
> https://github.com/edenhill/librdkafka/blob/master/INTRODUCTION.md#performance-numbers
> > > >
> > > > Do they correspond to your numbers?
> > > >
> > > > Consumer thruput is not included in that document but typically peaks
> > at
> > > > around 3 million msgs/s when run on the same host and with hot disk
> > > caches.
> > > >
> > > > Regards,
> > > > Magnus
> > > >
> > > >
> > > > 2014/1/10 Klaus Schaefers <kl...@ligatus.com>
> > > >
> > > > > Hi,
> > > > >
> > > > > I am currently benchmarking Kafka against ActiveMQ and I got some
> > > results
> > > > > the surprised my quite a lot. ActiveMQ managed to deliver 4x more
> > > > messages
> > > > > when running locally. But from all what I was reading I am a little
> > bit
> > > > > surprised. Honestly I expected Kafka to outperform ActiveMQ. Some I
> > am
> > > > left
> > > > > with the feeling that I configured some wrong or did any kind of
> > other
> > > > > mistake.
> > > > >
> > > > > My setup looks like this:
> > > > >
> > > > > - One local broker
> > > > > - 10 Topic / Queues
> > > > > - 1 producer that dispatches messages randomly to the topics
> > > > > - 1 consumer per topic
> > > > >
> > > > > I basically used the example on the Kafka web page.
> > > > >
> > > > > Also I encountered some issues when increasing the number of
> topics,
> > > lets
> > > > > say 100. In this case the consumer cannot connect the Zookeeper...
> > > > >
> > > > > Does anybody has an idea how to improve the performance?
> > > > >
> > > > >
> > > > > Thx,
> > > > >
> > > > > Klaus
> > > > >
> > > > >
> > > > > --
> > > > >
> > > > > --
> > > > >
> > > > > Klaus Schaefers
> > > > > Senior Optimization Manager
> > > > >
> > > > > Ligatus GmbH
> > > > > Hohenstaufenring 30-32
> > > > > D-50674 Köln
> > > > >
> > > > > Tel.:  +49 (0) 221 / 56939 -784
> > > > > Fax:  +49 (0) 221 / 56 939 - 599
> > > > > E-Mail: klaus.schaefers@ligatus.com
> > > > > Web: www.ligatus.de
> > > > >
> > > > > HRB Köln 56003
> > > > > Geschäftsführung:
> > > > > Dipl.-Kaufmann Lars Hasselbach, Dipl.-Kaufmann Klaus Ludemann,
> > > > > Dipl.-Wirtschaftsingenieur Arne Wolter
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > >
> > > --
> > >
> > > Klaus Schaefers
> > > Senior Optimization Manager
> > >
> > > Ligatus GmbH
> > > Hohenstaufenring 30-32
> > > D-50674 Köln
> > >
> > > Tel.:  +49 (0) 221 / 56939 -784
> > > Fax:  +49 (0) 221 / 56 939 - 599
> > > E-Mail: klaus.schaefers@ligatus.com
> > > Web: www.ligatus.de
> > >
> > > HRB Köln 56003
> > > Geschäftsführung:
> > > Dipl.-Kaufmann Lars Hasselbach, Dipl.-Kaufmann Klaus Ludemann,
> > > Dipl.-Wirtschaftsingenieur Arne Wolter
> > >
> >
>
>
>
> --
>
> --
>
> Klaus Schaefers
> Senior Optimization Manager
>
> Ligatus GmbH
> Hohenstaufenring 30-32
> D-50674 Köln
>
> Tel.:  +49 (0) 221 / 56939 -784
> Fax:  +49 (0) 221 / 56 939 - 599
> E-Mail: klaus.schaefers@ligatus.com
> Web: www.ligatus.de
>
> HRB Köln 56003
> Geschäftsführung:
> Dipl.-Kaufmann Lars Hasselbach, Dipl.-Kaufmann Klaus Ludemann,
> Dipl.-Wirtschaftsingenieur Arne Wolter
>

Re: Velocity on local machine

Posted by Klaus Schaefers <kl...@ligatus.com>.
Hi,

during my test cpu load is quite low, roughly 50 percent, sometimes peask
to 70%.

>Are you using the sync producer per chance?
I enforced now the async and I got a huge improvement in one of my test
cases. let my explore the rest a little bit more.

Cheers,

Klaus




On Fri, Jan 10, 2014 at 2:36 PM, Magnus Edenhill <ma...@edenhill.se> wrote:

> 2k msgs/s is silly, unless your messages are 10MB each, so something is
> clearly wrong.
>
> * What is the CPU usage and IO load when running your performance tests?
> * Are you using the sync producer per chance? Maybe combined with an
> aggressive log.flush.interval?
> * For reference, and to find out where the bottleneck is, try running
> rdkafka_performance [1] according to the previous link I posted:
>   if performance does not increase then the broker is the problem,
> otherwise it is the producer/consumers.
>
> Regards,
> Magnus
>
> [1]: https://github.com/edenhill/librdkafka/tree/master/examples
>
>
>
> 2014/1/10 Klaus Schaefers <kl...@ligatus.com>
>
> > Hi,
> >
> > I have close to 2k messages per second. My machine is just a (BG 4-core
> i5
> > but I would expect more messages. I ran Kafka in the default settings.
> >
> >
> > On Fri, Jan 10, 2014 at 12:31 PM, Magnus Edenhill <magnus@edenhill.se
> > >wrote:
> >
> > > What performance numbers did you see?
> > >
> > > For reference you can check the following tests that were also run on
> the
> > > same machine as the broker:
> > >
> > >
> >
> https://github.com/edenhill/librdkafka/blob/master/INTRODUCTION.md#performance-numbers
> > >
> > > Do they correspond to your numbers?
> > >
> > > Consumer thruput is not included in that document but typically peaks
> at
> > > around 3 million msgs/s when run on the same host and with hot disk
> > caches.
> > >
> > > Regards,
> > > Magnus
> > >
> > >
> > > 2014/1/10 Klaus Schaefers <kl...@ligatus.com>
> > >
> > > > Hi,
> > > >
> > > > I am currently benchmarking Kafka against ActiveMQ and I got some
> > results
> > > > the surprised my quite a lot. ActiveMQ managed to deliver 4x more
> > > messages
> > > > when running locally. But from all what I was reading I am a little
> bit
> > > > surprised. Honestly I expected Kafka to outperform ActiveMQ. Some I
> am
> > > left
> > > > with the feeling that I configured some wrong or did any kind of
> other
> > > > mistake.
> > > >
> > > > My setup looks like this:
> > > >
> > > > - One local broker
> > > > - 10 Topic / Queues
> > > > - 1 producer that dispatches messages randomly to the topics
> > > > - 1 consumer per topic
> > > >
> > > > I basically used the example on the Kafka web page.
> > > >
> > > > Also I encountered some issues when increasing the number of topics,
> > lets
> > > > say 100. In this case the consumer cannot connect the Zookeeper...
> > > >
> > > > Does anybody has an idea how to improve the performance?
> > > >
> > > >
> > > > Thx,
> > > >
> > > > Klaus
> > > >
> > > >
> > > > --
> > > >
> > > > --
> > > >
> > > > Klaus Schaefers
> > > > Senior Optimization Manager
> > > >
> > > > Ligatus GmbH
> > > > Hohenstaufenring 30-32
> > > > D-50674 Köln
> > > >
> > > > Tel.:  +49 (0) 221 / 56939 -784
> > > > Fax:  +49 (0) 221 / 56 939 - 599
> > > > E-Mail: klaus.schaefers@ligatus.com
> > > > Web: www.ligatus.de
> > > >
> > > > HRB Köln 56003
> > > > Geschäftsführung:
> > > > Dipl.-Kaufmann Lars Hasselbach, Dipl.-Kaufmann Klaus Ludemann,
> > > > Dipl.-Wirtschaftsingenieur Arne Wolter
> > > >
> > >
> >
> >
> >
> > --
> >
> > --
> >
> > Klaus Schaefers
> > Senior Optimization Manager
> >
> > Ligatus GmbH
> > Hohenstaufenring 30-32
> > D-50674 Köln
> >
> > Tel.:  +49 (0) 221 / 56939 -784
> > Fax:  +49 (0) 221 / 56 939 - 599
> > E-Mail: klaus.schaefers@ligatus.com
> > Web: www.ligatus.de
> >
> > HRB Köln 56003
> > Geschäftsführung:
> > Dipl.-Kaufmann Lars Hasselbach, Dipl.-Kaufmann Klaus Ludemann,
> > Dipl.-Wirtschaftsingenieur Arne Wolter
> >
>



-- 

-- 

Klaus Schaefers
Senior Optimization Manager

Ligatus GmbH
Hohenstaufenring 30-32
D-50674 Köln

Tel.:  +49 (0) 221 / 56939 -784
Fax:  +49 (0) 221 / 56 939 - 599
E-Mail: klaus.schaefers@ligatus.com
Web: www.ligatus.de

HRB Köln 56003
Geschäftsführung:
Dipl.-Kaufmann Lars Hasselbach, Dipl.-Kaufmann Klaus Ludemann,
Dipl.-Wirtschaftsingenieur Arne Wolter

Re: Velocity on local machine

Posted by Magnus Edenhill <ma...@edenhill.se>.
2k msgs/s is silly, unless your messages are 10MB each, so something is
clearly wrong.

* What is the CPU usage and IO load when running your performance tests?
* Are you using the sync producer per chance? Maybe combined with an
aggressive log.flush.interval?
* For reference, and to find out where the bottleneck is, try running
rdkafka_performance [1] according to the previous link I posted:
  if performance does not increase then the broker is the problem,
otherwise it is the producer/consumers.

Regards,
Magnus

[1]: https://github.com/edenhill/librdkafka/tree/master/examples



2014/1/10 Klaus Schaefers <kl...@ligatus.com>

> Hi,
>
> I have close to 2k messages per second. My machine is just a (BG 4-core i5
> but I would expect more messages. I ran Kafka in the default settings.
>
>
> On Fri, Jan 10, 2014 at 12:31 PM, Magnus Edenhill <magnus@edenhill.se
> >wrote:
>
> > What performance numbers did you see?
> >
> > For reference you can check the following tests that were also run on the
> > same machine as the broker:
> >
> >
> https://github.com/edenhill/librdkafka/blob/master/INTRODUCTION.md#performance-numbers
> >
> > Do they correspond to your numbers?
> >
> > Consumer thruput is not included in that document but typically peaks at
> > around 3 million msgs/s when run on the same host and with hot disk
> caches.
> >
> > Regards,
> > Magnus
> >
> >
> > 2014/1/10 Klaus Schaefers <kl...@ligatus.com>
> >
> > > Hi,
> > >
> > > I am currently benchmarking Kafka against ActiveMQ and I got some
> results
> > > the surprised my quite a lot. ActiveMQ managed to deliver 4x more
> > messages
> > > when running locally. But from all what I was reading I am a little bit
> > > surprised. Honestly I expected Kafka to outperform ActiveMQ. Some I am
> > left
> > > with the feeling that I configured some wrong or did any kind of other
> > > mistake.
> > >
> > > My setup looks like this:
> > >
> > > - One local broker
> > > - 10 Topic / Queues
> > > - 1 producer that dispatches messages randomly to the topics
> > > - 1 consumer per topic
> > >
> > > I basically used the example on the Kafka web page.
> > >
> > > Also I encountered some issues when increasing the number of topics,
> lets
> > > say 100. In this case the consumer cannot connect the Zookeeper...
> > >
> > > Does anybody has an idea how to improve the performance?
> > >
> > >
> > > Thx,
> > >
> > > Klaus
> > >
> > >
> > > --
> > >
> > > --
> > >
> > > Klaus Schaefers
> > > Senior Optimization Manager
> > >
> > > Ligatus GmbH
> > > Hohenstaufenring 30-32
> > > D-50674 Köln
> > >
> > > Tel.:  +49 (0) 221 / 56939 -784
> > > Fax:  +49 (0) 221 / 56 939 - 599
> > > E-Mail: klaus.schaefers@ligatus.com
> > > Web: www.ligatus.de
> > >
> > > HRB Köln 56003
> > > Geschäftsführung:
> > > Dipl.-Kaufmann Lars Hasselbach, Dipl.-Kaufmann Klaus Ludemann,
> > > Dipl.-Wirtschaftsingenieur Arne Wolter
> > >
> >
>
>
>
> --
>
> --
>
> Klaus Schaefers
> Senior Optimization Manager
>
> Ligatus GmbH
> Hohenstaufenring 30-32
> D-50674 Köln
>
> Tel.:  +49 (0) 221 / 56939 -784
> Fax:  +49 (0) 221 / 56 939 - 599
> E-Mail: klaus.schaefers@ligatus.com
> Web: www.ligatus.de
>
> HRB Köln 56003
> Geschäftsführung:
> Dipl.-Kaufmann Lars Hasselbach, Dipl.-Kaufmann Klaus Ludemann,
> Dipl.-Wirtschaftsingenieur Arne Wolter
>

Re: Velocity on local machine

Posted by Gerrit Jansen van Vuuren <ge...@gmail.com>.
Have you tried using more producers.
The kafka broker is performant, but the client producer's performance is
not what it should be.

You can also have a look at tuning the number of kafka broker's network and
io threads.


Regards,
 Gerrit


On Fri, Jan 10, 2014 at 1:06 PM, Klaus Schaefers <
klaus.schaefers@ligatus.com> wrote:

> Hi,
>
> I have close to 2k messages per second. My machine is just a (BG 4-core i5
> but I would expect more messages. I ran Kafka in the default settings.
>
>
> On Fri, Jan 10, 2014 at 12:31 PM, Magnus Edenhill <magnus@edenhill.se
> >wrote:
>
> > What performance numbers did you see?
> >
> > For reference you can check the following tests that were also run on the
> > same machine as the broker:
> >
> >
> https://github.com/edenhill/librdkafka/blob/master/INTRODUCTION.md#performance-numbers
> >
> > Do they correspond to your numbers?
> >
> > Consumer thruput is not included in that document but typically peaks at
> > around 3 million msgs/s when run on the same host and with hot disk
> caches.
> >
> > Regards,
> > Magnus
> >
> >
> > 2014/1/10 Klaus Schaefers <kl...@ligatus.com>
> >
> > > Hi,
> > >
> > > I am currently benchmarking Kafka against ActiveMQ and I got some
> results
> > > the surprised my quite a lot. ActiveMQ managed to deliver 4x more
> > messages
> > > when running locally. But from all what I was reading I am a little bit
> > > surprised. Honestly I expected Kafka to outperform ActiveMQ. Some I am
> > left
> > > with the feeling that I configured some wrong or did any kind of other
> > > mistake.
> > >
> > > My setup looks like this:
> > >
> > > - One local broker
> > > - 10 Topic / Queues
> > > - 1 producer that dispatches messages randomly to the topics
> > > - 1 consumer per topic
> > >
> > > I basically used the example on the Kafka web page.
> > >
> > > Also I encountered some issues when increasing the number of topics,
> lets
> > > say 100. In this case the consumer cannot connect the Zookeeper...
> > >
> > > Does anybody has an idea how to improve the performance?
> > >
> > >
> > > Thx,
> > >
> > > Klaus
> > >
> > >
> > > --
> > >
> > > --
> > >
> > > Klaus Schaefers
> > > Senior Optimization Manager
> > >
> > > Ligatus GmbH
> > > Hohenstaufenring 30-32
> > > D-50674 Köln
> > >
> > > Tel.:  +49 (0) 221 / 56939 -784
> > > Fax:  +49 (0) 221 / 56 939 - 599
> > > E-Mail: klaus.schaefers@ligatus.com
> > > Web: www.ligatus.de
> > >
> > > HRB Köln 56003
> > > Geschäftsführung:
> > > Dipl.-Kaufmann Lars Hasselbach, Dipl.-Kaufmann Klaus Ludemann,
> > > Dipl.-Wirtschaftsingenieur Arne Wolter
> > >
> >
>
>
>
> --
>
> --
>
> Klaus Schaefers
> Senior Optimization Manager
>
> Ligatus GmbH
> Hohenstaufenring 30-32
> D-50674 Köln
>
> Tel.:  +49 (0) 221 / 56939 -784
> Fax:  +49 (0) 221 / 56 939 - 599
> E-Mail: klaus.schaefers@ligatus.com
> Web: www.ligatus.de
>
> HRB Köln 56003
> Geschäftsführung:
> Dipl.-Kaufmann Lars Hasselbach, Dipl.-Kaufmann Klaus Ludemann,
> Dipl.-Wirtschaftsingenieur Arne Wolter
>

Re: Velocity on local machine

Posted by Klaus Schaefers <kl...@ligatus.com>.
Hi,

I have close to 2k messages per second. My machine is just a (BG 4-core i5
but I would expect more messages. I ran Kafka in the default settings.


On Fri, Jan 10, 2014 at 12:31 PM, Magnus Edenhill <ma...@edenhill.se>wrote:

> What performance numbers did you see?
>
> For reference you can check the following tests that were also run on the
> same machine as the broker:
>
> https://github.com/edenhill/librdkafka/blob/master/INTRODUCTION.md#performance-numbers
>
> Do they correspond to your numbers?
>
> Consumer thruput is not included in that document but typically peaks at
> around 3 million msgs/s when run on the same host and with hot disk caches.
>
> Regards,
> Magnus
>
>
> 2014/1/10 Klaus Schaefers <kl...@ligatus.com>
>
> > Hi,
> >
> > I am currently benchmarking Kafka against ActiveMQ and I got some results
> > the surprised my quite a lot. ActiveMQ managed to deliver 4x more
> messages
> > when running locally. But from all what I was reading I am a little bit
> > surprised. Honestly I expected Kafka to outperform ActiveMQ. Some I am
> left
> > with the feeling that I configured some wrong or did any kind of other
> > mistake.
> >
> > My setup looks like this:
> >
> > - One local broker
> > - 10 Topic / Queues
> > - 1 producer that dispatches messages randomly to the topics
> > - 1 consumer per topic
> >
> > I basically used the example on the Kafka web page.
> >
> > Also I encountered some issues when increasing the number of topics, lets
> > say 100. In this case the consumer cannot connect the Zookeeper...
> >
> > Does anybody has an idea how to improve the performance?
> >
> >
> > Thx,
> >
> > Klaus
> >
> >
> > --
> >
> > --
> >
> > Klaus Schaefers
> > Senior Optimization Manager
> >
> > Ligatus GmbH
> > Hohenstaufenring 30-32
> > D-50674 Köln
> >
> > Tel.:  +49 (0) 221 / 56939 -784
> > Fax:  +49 (0) 221 / 56 939 - 599
> > E-Mail: klaus.schaefers@ligatus.com
> > Web: www.ligatus.de
> >
> > HRB Köln 56003
> > Geschäftsführung:
> > Dipl.-Kaufmann Lars Hasselbach, Dipl.-Kaufmann Klaus Ludemann,
> > Dipl.-Wirtschaftsingenieur Arne Wolter
> >
>



-- 

-- 

Klaus Schaefers
Senior Optimization Manager

Ligatus GmbH
Hohenstaufenring 30-32
D-50674 Köln

Tel.:  +49 (0) 221 / 56939 -784
Fax:  +49 (0) 221 / 56 939 - 599
E-Mail: klaus.schaefers@ligatus.com
Web: www.ligatus.de

HRB Köln 56003
Geschäftsführung:
Dipl.-Kaufmann Lars Hasselbach, Dipl.-Kaufmann Klaus Ludemann,
Dipl.-Wirtschaftsingenieur Arne Wolter

Re: Velocity on local machine

Posted by Magnus Edenhill <ma...@edenhill.se>.
What performance numbers did you see?

For reference you can check the following tests that were also run on the
same machine as the broker:
https://github.com/edenhill/librdkafka/blob/master/INTRODUCTION.md#performance-numbers

Do they correspond to your numbers?

Consumer thruput is not included in that document but typically peaks at
around 3 million msgs/s when run on the same host and with hot disk caches.

Regards,
Magnus


2014/1/10 Klaus Schaefers <kl...@ligatus.com>

> Hi,
>
> I am currently benchmarking Kafka against ActiveMQ and I got some results
> the surprised my quite a lot. ActiveMQ managed to deliver 4x more messages
> when running locally. But from all what I was reading I am a little bit
> surprised. Honestly I expected Kafka to outperform ActiveMQ. Some I am left
> with the feeling that I configured some wrong or did any kind of other
> mistake.
>
> My setup looks like this:
>
> - One local broker
> - 10 Topic / Queues
> - 1 producer that dispatches messages randomly to the topics
> - 1 consumer per topic
>
> I basically used the example on the Kafka web page.
>
> Also I encountered some issues when increasing the number of topics, lets
> say 100. In this case the consumer cannot connect the Zookeeper...
>
> Does anybody has an idea how to improve the performance?
>
>
> Thx,
>
> Klaus
>
>
> --
>
> --
>
> Klaus Schaefers
> Senior Optimization Manager
>
> Ligatus GmbH
> Hohenstaufenring 30-32
> D-50674 Köln
>
> Tel.:  +49 (0) 221 / 56939 -784
> Fax:  +49 (0) 221 / 56 939 - 599
> E-Mail: klaus.schaefers@ligatus.com
> Web: www.ligatus.de
>
> HRB Köln 56003
> Geschäftsführung:
> Dipl.-Kaufmann Lars Hasselbach, Dipl.-Kaufmann Klaus Ludemann,
> Dipl.-Wirtschaftsingenieur Arne Wolter
>