You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by Brian Crowell <br...@fluggo.com> on 2010/07/26 22:18:15 UTC
QPID message throughput - Red Hat numbers
Red Hat claims to be able to get hundreds of thousands of messages
through on an eight core machine
(http://www.redhat.com/mrg/messaging/features/ or
http://www.redhat.com/f/pdf/mrg/Reference_Architecture_MRG_Messaging_Throughput.pdf).
I'm working with an eight-core machine, and I'm only getting about
11,000/sec (in; about 6,500/sec out). This is with perftest, default
settings.
What kinds of things do I need to be doing to get better throughput?
--Brian
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: QPID message throughput - Red Hat numbers
Posted by Brian Crowell <br...@fluggo.com>.
On Wed, Jul 28, 2010 at 12:50 PM, Jonathan Robie
<jo...@redhat.com> wrote:
> Would a last value queue be helpful here?
Good suggestion, but no. I'm calculating VWAPs.
--Brian
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: QPID message throughput - Red Hat numbers
Posted by Jonathan Robie <jo...@redhat.com>.
On 07/28/2010 01:28 PM, Brian Crowell wrote:
> It sounds like I'm just going to have to create a more complicated
> setup than I hoped for, trying to divide all the messages up into as
> many queues as I can. I originally wanted all messages to go into a
> few exchanges and let the subscriber bind to whichever stock it wanted
> to know about.
Would a last value queue be helpful here?
Jonathan
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: QPID message throughput - Red Hat numbers
Posted by Gordon Sim <gs...@redhat.com>.
On 07/28/2010 06:55 PM, Brian Crowell wrote:
> On Wed, Jul 28, 2010 at 12:34 PM, Gordon Sim<gs...@redhat.com> wrote:
>> What are your settings for the subscriber? Try batching (or not requiring)
>> acks. I wouldn't expect a single producer to be overwhelmingly faster than a
>> subscriber.
>
> I'm using the Qpid C++ client API. My accept mode is explicit (so I
> don't receive more messages than I can process; this isn't really much
> of a problem anymore, as I've hit Qpid's limit now). Messages are
> pre-acquired, there's a window of 1000 messages, and autoAck is 5. I'm
> using the SubscriptionManager with a callback.
>
> With these settings, my receive rate comes pretty close to what I can
> get with perftest. I'm experimenting with scheduling priorities and
> CPU affinities to see what kind of improvements I can get.
11k is very slow, if you can get a linux box with more cores I would
expect that to go up significantly.
>> So the subscriber isn't actually interested in all the messages coming in?
>> Again that should make a difference as the subscriber rate doesn't need to
>> match the total publisher rate.
>
> This *particular* subscriber is interested in 99.9% of what the
> publishers push. Other subscribers will bind to specific topics.
Ok, so the rate for that subscriber will be the limiting factor in the
overall system. Is there any opportunity at all for somehow splitting
that subscribers job up?
>> Sessions won't really improve things; the unit of parallelism on the broker
>> is at the connection level. Each connection is processed by a single
>> thread/core at a time.
>
> Interesting, I thought I had read the opposite somewhere. So sessions
> are mainly for picking up where you left off?
Sessions are for multiplexing different logical streams over a single
connection. On the *client* you can have different sessions processed by
different threads which can give you some parallelism, but on the server
its at the connection level only.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: QPID message throughput - Red Hat numbers
Posted by Brian Crowell <br...@fluggo.com>.
On Wed, Jul 28, 2010 at 12:34 PM, Gordon Sim <gs...@redhat.com> wrote:
> What are your settings for the subscriber? Try batching (or not requiring)
> acks. I wouldn't expect a single producer to be overwhelmingly faster than a
> subscriber.
I'm using the Qpid C++ client API. My accept mode is explicit (so I
don't receive more messages than I can process; this isn't really much
of a problem anymore, as I've hit Qpid's limit now). Messages are
pre-acquired, there's a window of 1000 messages, and autoAck is 5. I'm
using the SubscriptionManager with a callback.
With these settings, my receive rate comes pretty close to what I can
get with perftest. I'm experimenting with scheduling priorities and
CPU affinities to see what kind of improvements I can get.
> So the subscriber isn't actually interested in all the messages coming in?
> Again that should make a difference as the subscriber rate doesn't need to
> match the total publisher rate.
This *particular* subscriber is interested in 99.9% of what the
publishers push. Other subscribers will bind to specific topics.
> Sessions won't really improve things; the unit of parallelism on the broker
> is at the connection level. Each connection is processed by a single
> thread/core at a time.
Interesting, I thought I had read the opposite somewhere. So sessions
are mainly for picking up where you left off?
--Brian
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: QPID message throughput - Red Hat numbers
Posted by Gordon Sim <gs...@redhat.com>.
On 07/28/2010 06:28 PM, Brian Crowell wrote:
> On Wed, Jul 28, 2010 at 12:01 PM, Gordon Sim<gs...@redhat.com> wrote:
>> If you have four connections pumping in messages to a queue as fast as they
>> can and only one pulling them out, then the queue will indeed backup.
>
> The best part is that only one publisher is really pushing as fast as
> it can at any one time. One publisher is enough to overwhelm the
> subscriber.
What are your settings for the subscriber? Try batching (or not
requiring) acks. I wouldn't expect a single producer to be
overwhelmingly faster than a subscriber.
>>> Messages pile up in the queue
>>> and I run out of memory fast. I suppose for that, I'll have to devote
>>> one queue to each publisher.
>>
>> Or perhaps build in some feedback/throttling into the application(?).
>
> Funny thing: I'm actually using Qpid for sending around stock ticks,
> which they say they designed AMQP to handle. So I can't slow down the
> messages that come in. If I can't send them into Qpid fast enough,
> they pile up at the publisher. If I can't read them out of Qpid fast
> enough, they pile up in Qpid.
>
> It sounds like I'm just going to have to create a more complicated
> setup than I hoped for, trying to divide all the messages up into as
> many queues as I can. I originally wanted all messages to go into a
> few exchanges and let the subscriber bind to whichever stock it wanted
> to know about.
So the subscriber isn't actually interested in all the messages coming
in? Again that should make a difference as the subscriber rate doesn't
need to match the total publisher rate.
> So:
>
> * More sessions at the publisher so it can feed more ticks.
Sessions won't really improve things; the unit of parallelism on the
broker is at the connection level. Each connection is processed by a
single thread/core at a time.
> * As many queues as I can manage at the subscriber.
> * Qpid on Linux with RT kernel.
>
> --Brian
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: QPID message throughput - Red Hat numbers
Posted by Brian Crowell <br...@fluggo.com>.
On Wed, Jul 28, 2010 at 12:01 PM, Gordon Sim <gs...@redhat.com> wrote:
> If you have four connections pumping in messages to a queue as fast as they
> can and only one pulling them out, then the queue will indeed backup.
The best part is that only one publisher is really pushing as fast as
it can at any one time. One publisher is enough to overwhelm the
subscriber.
>> Messages pile up in the queue
>> and I run out of memory fast. I suppose for that, I'll have to devote
>> one queue to each publisher.
>
> Or perhaps build in some feedback/throttling into the application(?).
Funny thing: I'm actually using Qpid for sending around stock ticks,
which they say they designed AMQP to handle. So I can't slow down the
messages that come in. If I can't send them into Qpid fast enough,
they pile up at the publisher. If I can't read them out of Qpid fast
enough, they pile up in Qpid.
It sounds like I'm just going to have to create a more complicated
setup than I hoped for, trying to divide all the messages up into as
many queues as I can. I originally wanted all messages to go into a
few exchanges and let the subscriber bind to whichever stock it wanted
to know about.
So:
* More sessions at the publisher so it can feed more ticks.
* As many queues as I can manage at the subscriber.
* Qpid on Linux with RT kernel.
--Brian
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: QPID message throughput - Red Hat numbers
Posted by Gordon Sim <gs...@redhat.com>.
On 07/28/2010 05:43 PM, Brian Crowell wrote:
> On Wed, Jul 28, 2010 at 5:15 AM, Gordon Sim<gs...@redhat.com> wrote:
>> Are you running the same test scenario as described in 3.3 of that document?
>> I.e. Simulating "60 AMQP clients talking to the AMQP broker with 10 shared
>> queues". (That is not what you get for perftest with 'default settings'
>> which is why I ask what may be a stupid question).
>
> I'm not. I wanted to start by trying to get a baseline stat for one
> queue (that is, how fast can Qpid serve one session?). The actual
> scenario I'm working with is closer to:
>
> ./perftest --mode topic --count 100000 --npubs 4 --size 100
> --pub-confirm no -s
>
> I had installed on a Windows box, but I'm comparing it to a Linux box
> now, and the Linux box is trouncing it. From the comments here, it
> sounds like I'll need to go Linux if I ever hope to get any
> performance out of it. I had hoped there was something simple I wasn't
> doing or some build option I could change, but it sounds like the OS
> is a major factor.
>
> On the above test:
>
> Windows 2003 (8-core)
> Pubs: 5671 msg/sec
> Subs: 8672 msg/sec
> Total: 17262 msg/sec
> Throughput: 1.64 MiB/s
>
> Debian Linux (1-core?)
> Pubs: 5892 msg/sec
> Subs: 11343 msg/sec
> Total: 22687 msg/sec
> Throughput: 2.16 MiB/s
>
> The other thing I'm worried about is that I can push so much faster
> *into* Qpid than I can pull out of it.
If you have four connections pumping in messages to a queue as fast as
they can and only one pulling them out, then the queue will indeed
backup. I am very keen to get producer flow control implemented which
would automatically slow the producers down to a rate that matched the
consumer(s) in the steady state.
> Messages pile up in the queue
> and I run out of memory fast. I suppose for that, I'll have to devote
> one queue to each publisher.
Or perhaps build in some feedback/throttling into the application(?).
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: QPID message throughput - Red Hat numbers
Posted by Rajith Attapattu <ra...@gmail.com>.
On Wed, Jul 28, 2010 at 12:43 PM, Brian Crowell <br...@fluggo.com> wrote:
> On Wed, Jul 28, 2010 at 5:15 AM, Gordon Sim <gs...@redhat.com> wrote:
>> Are you running the same test scenario as described in 3.3 of that document?
>> I.e. Simulating "60 AMQP clients talking to the AMQP broker with 10 shared
>> queues". (That is not what you get for perftest with 'default settings'
>> which is why I ask what may be a stupid question).
>
> I'm not. I wanted to start by trying to get a baseline stat for one
> queue (that is, how fast can Qpid serve one session?). The actual
> scenario I'm working with is closer to:
>
> ./perftest --mode topic --count 100000 --npubs 4 --size 100
> --pub-confirm no -s
>
> I had installed on a Windows box, but I'm comparing it to a Linux box
> now, and the Linux box is trouncing it. From the comments here, it
> sounds like I'll need to go Linux if I ever hope to get any
> performance out of it. I had hoped there was something simple I wasn't
> doing or some build option I could change, but it sounds like the OS
> is a major factor.
>
> On the above test:
>
> Windows 2003 (8-core)
> Pubs: 5671 msg/sec
> Subs: 8672 msg/sec
> Total: 17262 msg/sec
> Throughput: 1.64 MiB/s
>
> Debian Linux (1-core?)
> Pubs: 5892 msg/sec
> Subs: 11343 msg/sec
> Total: 22687 msg/sec
> Throughput: 2.16 MiB/s
>
> The other thing I'm worried about is that I can push so much faster
> *into* Qpid than I can pull out of it. Messages pile up in the queue
> and I run out of memory fast. I suppose for that, I'll have to devote
> one queue to each publisher.
You could set max size (in terms of message or bytes) when you create a queue.
You could also set a queue policy to tell the broker what it should do
if this limit is reached.
The default is "reject", which will throw an exception to the
publisher saying the queue is full.
( Other options are "ring" and "flow-to-disk" and the latter is
obviously very slow).
Rajith
> --Brian
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>
>
--
Regards,
Rajith Attapattu
Red Hat
http://rajith.2rlabs.com/
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
RE: QPID message throughput - Red Hat numbers
Posted by Steve Huston <sh...@riverace.com>.
Hi Brian,
> On Wed, Jul 28, 2010 at 5:15 AM, Gordon Sim <gs...@redhat.com> wrote:
> > Are you running the same test scenario as described in 3.3 of that
> > document? I.e. Simulating "60 AMQP clients talking to the
> AMQP broker
> > with 10 shared queues". (That is not what you get for perftest with
> > 'default settings' which is why I ask what may be a stupid
> question).
>
> I'm not. I wanted to start by trying to get a baseline stat
> for one queue (that is, how fast can Qpid serve one
> session?). The actual scenario I'm working with is closer to:
>
> ./perftest --mode topic --count 100000 --npubs 4 --size 100
> --pub-confirm no -s
>
> I had installed on a Windows box, but I'm comparing it to a
> Linux box now, and the Linux box is trouncing it. From the
> comments here, it sounds like I'll need to go Linux if I ever
> hope to get any performance out of it.
Just a note... I worked on the Windows port and I can tell you we never
tuned the performance, analyzed bottlenecks, etc. I don't know if a
small amount of such analysis and tuning could get much improved
performance, but it's a possibility you may want to consider in the
grand scheme of things.
> I had hoped there was
> something simple I wasn't doing or some build option I could
> change, but it sounds like the OS is a major factor.
>
> On the above test:
>
> Windows 2003 (8-core)
> Pubs: 5671 msg/sec
> Subs: 8672 msg/sec
> Total: 17262 msg/sec
> Throughput: 1.64 MiB/s
>
> Debian Linux (1-core?)
> Pubs: 5892 msg/sec
> Subs: 11343 msg/sec
> Total: 22687 msg/sec
> Throughput: 2.16 MiB/s
Oh my - that is bad. I'm not a Windows bigot but I'd think that it
should be closer than that.
-Steve
--
Steve Huston, Riverace Corporation
Total Lifecycle Support for Your Networked Applications
http://www.riverace.com
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: QPID message throughput - Red Hat numbers
Posted by Brian Crowell <br...@fluggo.com>.
On Wed, Jul 28, 2010 at 5:15 AM, Gordon Sim <gs...@redhat.com> wrote:
> Are you running the same test scenario as described in 3.3 of that document?
> I.e. Simulating "60 AMQP clients talking to the AMQP broker with 10 shared
> queues". (That is not what you get for perftest with 'default settings'
> which is why I ask what may be a stupid question).
I'm not. I wanted to start by trying to get a baseline stat for one
queue (that is, how fast can Qpid serve one session?). The actual
scenario I'm working with is closer to:
./perftest --mode topic --count 100000 --npubs 4 --size 100
--pub-confirm no -s
I had installed on a Windows box, but I'm comparing it to a Linux box
now, and the Linux box is trouncing it. From the comments here, it
sounds like I'll need to go Linux if I ever hope to get any
performance out of it. I had hoped there was something simple I wasn't
doing or some build option I could change, but it sounds like the OS
is a major factor.
On the above test:
Windows 2003 (8-core)
Pubs: 5671 msg/sec
Subs: 8672 msg/sec
Total: 17262 msg/sec
Throughput: 1.64 MiB/s
Debian Linux (1-core?)
Pubs: 5892 msg/sec
Subs: 11343 msg/sec
Total: 22687 msg/sec
Throughput: 2.16 MiB/s
The other thing I'm worried about is that I can push so much faster
*into* Qpid than I can pull out of it. Messages pile up in the queue
and I run out of memory fast. I suppose for that, I'll have to devote
one queue to each publisher.
--Brian
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: QPID message throughput - Red Hat numbers
Posted by Gordon Sim <gs...@redhat.com>.
On 07/26/2010 09:18 PM, Brian Crowell wrote:
> Red Hat claims to be able to get hundreds of thousands of messages
> through on an eight core machine
> (http://www.redhat.com/mrg/messaging/features/ or
> http://www.redhat.com/f/pdf/mrg/Reference_Architecture_MRG_Messaging_Throughput.pdf).
> I'm working with an eight-core machine, and I'm only getting about
> 11,000/sec (in; about 6,500/sec out). This is with perftest, default
> settings.
>
> What kinds of things do I need to be doing to get better throughput?
Are you running the same test scenario as described in 3.3 of that
document? I.e. Simulating "60 AMQP clients talking to the AMQP broker
with 10 shared queues". (That is not what you get for perftest with
'default settings' which is why I ask what may be a stupid question).
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: QPID message throughput - Red Hat numbers
Posted by Alan Conway <ac...@redhat.com>.
On 07/27/2010 10:44 AM, Carl Trieloff wrote:
>
> The boxes are then tuned according to the tuning guide, see
> http://www.redhat.com/mrg/resources/
You're referring to the Realtime Tuning guide here?
> Note that none of the test use the onboard NIC, onboard NIC's are
> usually built
> to save $ and never yield the best results. On most of the slides that I
> have used
> the full machine config is listed, or most of the MRG papers / presso's
> have the
> tunings, and options used on the tests to reproduce in the appendix or
> last few
> slides of the deck
>
> If you are using the RPMs and not getting comp numbers, it is usually
> setting --worker-threads,
> SMI's, Memory speed (or overpopulated/unbalanced), NIC, and tuning --
> and in that order.
Do we have a public tuning guide for qpid that explains those steps?
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: QPID message throughput - Red Hat numbers
Posted by Carl Trieloff <cc...@redhat.com>.
Tests from MRG rpms are done as follow:
perftest results are transfers per second, as that is what perftest reports
latency test are messages per second as that is what latency test reports.
Note that the 33% increase is an old paper, pre RHEL 5.4 note that RHEL5.4
now has an updated memory allocator based on this work and now beats the
optimized memory allocator numbers from that paper.
You can also set thread affinity on the allocator in RHEL5.4 which in some
cases gives you an increase on NUMA machine. Note that RHEL6 now does
this by default for NUMA, and in the numbers Mark Wagner & I showed at
Red Hat Summit indicate ~ 10% gain from RHEL 5.x to RHEL 6 beta 2, snap7.
Test are done on RT certified machines,
see
https://hardware.redhat.com/list.cgi?version=5&field0-0-0=cf_fixed_in&type0-0-0=substring&value0-0-0=MRG
or on whiteboxes.
all tests are done with SMI's turned off.
The boxes are then tuned according to the tuning guide, see
http://www.redhat.com/mrg/resources/
Note that none of the test use the onboard NIC, onboard NIC's are
usually built
to save $ and never yield the best results. On most of the slides that I
have used
the full machine config is listed, or most of the MRG papers / presso's
have the
tunings, and options used on the tests to reproduce in the appendix or
last few
slides of the deck
If you are using the RPMs and not getting comp numbers, it is usually
setting --worker-threads,
SMI's, Memory speed (or overpopulated/unbalanced), NIC, and tuning --
and in that order.
What has been great is with the published numbers is that people try to
reproduce then,
in many cases they reproduce. when they don't it gets questions being
asked and serves
as a great tool to debug the environment/ network. It has been amazing
the things that
have been found. In all cases I've been involved in it has resulted in
the numbers being
reproduced within a % or so.
hope that helps
Carl.
On 07/26/2010 10:06 PM, Clark O'Brien wrote:
> Is possible to replicate this environment in some cloud that supports pay-per-use like Amazon? I would live to test this in an environment that I did not have to tear down.
>
>
>
> --- On Mon, 7/26/10, Donohue, Matt<md...@structure-tech.com> wrote:
>
>
>> From: Donohue, Matt<md...@structure-tech.com>
>> Subject: RE: QPID message throughput - Red Hat numbers
>> To: "Clark O'Brien"<cl...@yahoo.com>, "users@qpid.apache.org"<us...@qpid.apache.org>, "brian@fluggo.com"<br...@fluggo.com>
>> Date: Monday, July 26, 2010, 7:49 PM
>>
>> This was an Intel Xeon 5570 box with the RT kernel and
>> following the Messaging install and optimizations
>> recommended by RH's MRG docs.
>> http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_MRG/
>>
>> I dug up the numbers I saw-
>>
>> Running-
>> ./perftest --port 40000 --username guest --password guest
>> --tcp-nodelay --size 256 --npubs 10 --pub-confirm no --mode
>> hared --async-commit yes -s
>>
>> Summary output:
>> pubs/sec subs/sec transfers/sec Mbytes/sec
>>
>> (rh qpid rpm and tcp-nodelay on)
>> 28924.2 49230.9 98464.3 24.0391
>>
>> (rh qpid rpm and tcp-nodelay off)
>> 12227 20663.9 41341.9 10.0932
>>
>> (compiled qpidc-0.5 and tcp-nodelay on)
>> 13523 21557 43122 10.5278
>>
>>
>> -----Original Message-----
>> From: Clark O'Brien [mailto:clark_obrien@yahoo.com]
>>
>> Sent: Monday, July 26, 2010 8:08 PM
>> To: users@qpid.apache.org;
>> Donohue, Matt; brian@fluggo.com
>> Subject: Re: QPID message throughput - Red Hat numbers
>>
>> A couple of interesting comments from the red hat doc.
>>
>>
>> The Intel® Xeon® 5482 based system increases throughput
>> by ~ 48% over the Intel® Xeon® 5365 based system. [On
>> average, 344K messages/sec for the Intel® Xeon® 5365 based
>> system versus 505K messages/sec for the Intel® Xeon® 5482
>> based system.]
>>
>> The optimized memory allocator increased throughput by ~
>> 49.7%. [On average, 558 messages/sec for the Intel® Xeon®
>> 5365 based system versus 762K messages/sec for the Intel®
>> Xeon® 5482 based system. Intel® Xeon® 5482 based system
>> increased throughput by 36.6%.]
>> --- On Mon, 7/26/10, Ian.Kinkade<ki...@idi-middleware.com>
>> wrote:
>>
>>
>>> From: Ian.Kinkade<ki...@idi-middleware.com>
>>> Subject: Re: QPID message throughput - Red Hat
>>>
>> numbers
>>
>>> To: users@qpid.apache.org,
>>>
>> mdonohue@structure-tech.com,
>> brian@fluggo.com
>>
>>> Date: Monday, July 26, 2010, 6:30 PM
>>> Hi Matt& Brian,
>>>
>>> It is my understanding that the Red Hat tests were
>>> conducted using a
>>> Real-time Version of RHEL (MRG) and that it was
>>> specifically tuned for
>>> MRG-M and its test applications.
>>>
>>> You might want to try using the tuning application
>>>
>> from the
>>
>>> MRG install
>>> before you run the tests.
>>>
>>> I hope this was helpful?
>>>
>>> Best Regards .................... Ian
>>>
>>> Ian Kinkade
>>> CEO
>>> Information Design, Inc.
>>> 145 Durham Road, Suite 11
>>> Madison, CT 06443 USA
>>> URL: www.idi-middleware.com
>>> Email: kinkade@idi-middleware.com
>>>
>>> Work: 203-245-0772 Ext: 6212
>>> Fax: 203-245-1885
>>> Cell: 203-589-1192
>>>
>>>
>>> On 7/26/2010 7:54 PM, Donohue, Matt wrote:
>>>
>>>> The last project I worked on was the same for me.
>>>>
>> Not
>>
>>> close to the MRG throughput numbers with the same test
>>>
>> and
>>
>>> this was on an otherwise optimized trading box.
>>>
>>>> The MRG qpid rpm was faster than an Intel C++
>>>>
>> compiled
>>
>>> version though.
>>>
>>>> Regards,
>>>> Matt
>>>>
>>>> -----Original Message-----
>>>> From: Brian Crowell [mailto:brian@fluggo.com]
>>>> Sent: Monday, July 26, 2010 3:18 PM
>>>> To: users@qpid.apache.org
>>>> Subject: QPID message throughput - Red Hat
>>>>
>> numbers
>>
>>>> Red Hat claims to be able to get hundreds of
>>>>
>> thousands
>>
>>> of messages
>>>
>>>> through on an eight core machine
>>>> (http://www.redhat.com/mrg/messaging/features/ or
>>>> http://www.redhat.com/f/pdf/mrg/Reference_Architecture_MRG_Messaging_Throughput.pdf).
>>>> I'm working with an eight-core machine, and I'm
>>>>
>> only
>>
>>> getting about
>>>
>>>> 11,000/sec (in; about 6,500/sec out). This is
>>>>
>> with
>>
>>> perftest, default
>>>
>>>> settings.
>>>>
>>>> What kinds of things do I need to be doing to
>>>>
>> get
>>
>>> better throughput?
>>>
>>>> --Brian
>>>>
>>>>
>>>>
>>>
>> ---------------------------------------------------------------------
>>
>>>> Apache Qpid - AMQP Messaging Implementation
>>>> Project: http://qpid.apache.org
>>>> Use/Interact: mailto:users-subscribe@qpid.apache.org
>>>>
>>>>
>>>>
>>>>
>>>
>> ---------------------------------------------------------------------
>>
>>>> Apache Qpid - AMQP Messaging Implementation
>>>> Project: http://qpid.apache.org
>>>> Use/Interact: mailto:users-subscribe@qpid.apache.org
>>>>
>>>>
>>>>
>>>
>>>
>> ---------------------------------------------------------------------
>>
>>> Apache Qpid - AMQP Messaging Implementation
>>> Project: http://qpid.apache.org
>>> Use/Interact: mailto:users-subscribe@qpid.apache.org
>>>
>>>
>>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> Apache Qpid - AMQP Messaging Implementation
>> Project: http://qpid.apache.org
>> Use/Interact: mailto:users-subscribe@qpid.apache.org
>>
>>
>>
>
>
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>
>
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
RE: QPID message throughput - Red Hat numbers
Posted by Clark O'Brien <cl...@yahoo.com>.
Is possible to replicate this environment in some cloud that supports pay-per-use like Amazon? I would live to test this in an environment that I did not have to tear down.
--- On Mon, 7/26/10, Donohue, Matt <md...@structure-tech.com> wrote:
> From: Donohue, Matt <md...@structure-tech.com>
> Subject: RE: QPID message throughput - Red Hat numbers
> To: "Clark O'Brien" <cl...@yahoo.com>, "users@qpid.apache.org" <us...@qpid.apache.org>, "brian@fluggo.com" <br...@fluggo.com>
> Date: Monday, July 26, 2010, 7:49 PM
>
> This was an Intel Xeon 5570 box with the RT kernel and
> following the Messaging install and optimizations
> recommended by RH's MRG docs.
> http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_MRG/
>
> I dug up the numbers I saw-
>
> Running-
> ./perftest --port 40000 --username guest --password guest
> --tcp-nodelay --size 256 --npubs 10 --pub-confirm no --mode
> hared --async-commit yes -s
>
> Summary output:
> pubs/sec subs/sec transfers/sec Mbytes/sec
>
> (rh qpid rpm and tcp-nodelay on)
> 28924.2 49230.9 98464.3 24.0391
>
> (rh qpid rpm and tcp-nodelay off)
> 12227 20663.9 41341.9 10.0932
>
> (compiled qpidc-0.5 and tcp-nodelay on)
> 13523 21557 43122 10.5278
>
>
> -----Original Message-----
> From: Clark O'Brien [mailto:clark_obrien@yahoo.com]
>
> Sent: Monday, July 26, 2010 8:08 PM
> To: users@qpid.apache.org;
> Donohue, Matt; brian@fluggo.com
> Subject: Re: QPID message throughput - Red Hat numbers
>
> A couple of interesting comments from the red hat doc.
>
>
> The Intel® Xeon® 5482 based system increases throughput
> by ~ 48% over the Intel® Xeon® 5365 based system. [On
> average, 344K messages/sec for the Intel® Xeon® 5365 based
> system versus 505K messages/sec for the Intel® Xeon® 5482
> based system.]
>
> The optimized memory allocator increased throughput by ~
> 49.7%. [On average, 558 messages/sec for the Intel® Xeon®
> 5365 based system versus 762K messages/sec for the Intel®
> Xeon® 5482 based system. Intel® Xeon® 5482 based system
> increased throughput by 36.6%.]
> --- On Mon, 7/26/10, Ian.Kinkade <ki...@idi-middleware.com>
> wrote:
>
> > From: Ian.Kinkade <ki...@idi-middleware.com>
> > Subject: Re: QPID message throughput - Red Hat
> numbers
> > To: users@qpid.apache.org,
> mdonohue@structure-tech.com,
> brian@fluggo.com
> > Date: Monday, July 26, 2010, 6:30 PM
> > Hi Matt & Brian,
> >
> > It is my understanding that the Red Hat tests were
> > conducted using a
> > Real-time Version of RHEL (MRG) and that it was
> > specifically tuned for
> > MRG-M and its test applications.
> >
> > You might want to try using the tuning application
> from the
> > MRG install
> > before you run the tests.
> >
> > I hope this was helpful?
> >
> > Best Regards .................... Ian
> >
> > Ian Kinkade
> > CEO
> > Information Design, Inc.
> > 145 Durham Road, Suite 11
> > Madison, CT 06443 USA
> > URL: www.idi-middleware.com
> > Email: kinkade@idi-middleware.com
> >
> > Work: 203-245-0772 Ext: 6212
> > Fax: 203-245-1885
> > Cell: 203-589-1192
> >
> >
> > On 7/26/2010 7:54 PM, Donohue, Matt wrote:
> > > The last project I worked on was the same for me.
> Not
> > close to the MRG throughput numbers with the same test
> and
> > this was on an otherwise optimized trading box.
> > > The MRG qpid rpm was faster than an Intel C++
> compiled
> > version though.
> > >
> > > Regards,
> > > Matt
> > >
> > > -----Original Message-----
> > > From: Brian Crowell [mailto:brian@fluggo.com]
> > > Sent: Monday, July 26, 2010 3:18 PM
> > > To: users@qpid.apache.org
> > > Subject: QPID message throughput - Red Hat
> numbers
> > >
> > > Red Hat claims to be able to get hundreds of
> thousands
> > of messages
> > > through on an eight core machine
> > > (http://www.redhat.com/mrg/messaging/features/ or
> > > http://www.redhat.com/f/pdf/mrg/Reference_Architecture_MRG_Messaging_Throughput.pdf).
> > > I'm working with an eight-core machine, and I'm
> only
> > getting about
> > > 11,000/sec (in; about 6,500/sec out). This is
> with
> > perftest, default
> > > settings.
> > >
> > > What kinds of things do I need to be doing to
> get
> > better throughput?
> > >
> > > --Brian
> > >
> > >
> >
> ---------------------------------------------------------------------
> > > Apache Qpid - AMQP Messaging Implementation
> > > Project: http://qpid.apache.org
> > > Use/Interact: mailto:users-subscribe@qpid.apache.org
> > >
> > >
> > >
> >
> ---------------------------------------------------------------------
> > > Apache Qpid - AMQP Messaging Implementation
> > > Project: http://qpid.apache.org
> > > Use/Interact: mailto:users-subscribe@qpid.apache.org
> > >
> > >
> >
> >
> ---------------------------------------------------------------------
> > Apache Qpid - AMQP Messaging Implementation
> > Project: http://qpid.apache.org
> > Use/Interact: mailto:users-subscribe@qpid.apache.org
> >
> >
>
>
>
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>
>
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
RE: QPID message throughput - Red Hat numbers
Posted by "Donohue, Matt" <md...@structure-tech.com>.
This was an Intel Xeon 5570 box with the RT kernel and following the Messaging install and optimizations recommended by RH's MRG docs.
http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_MRG/
I dug up the numbers I saw-
Running-
./perftest --port 40000 --username guest --password guest --tcp-nodelay --size 256 --npubs 10 --pub-confirm no --mode hared --async-commit yes -s
Summary output:
pubs/sec subs/sec transfers/sec Mbytes/sec
(rh qpid rpm and tcp-nodelay on)
28924.2 49230.9 98464.3 24.0391
(rh qpid rpm and tcp-nodelay off)
12227 20663.9 41341.9 10.0932
(compiled qpidc-0.5 and tcp-nodelay on)
13523 21557 43122 10.5278
-----Original Message-----
From: Clark O'Brien [mailto:clark_obrien@yahoo.com]
Sent: Monday, July 26, 2010 8:08 PM
To: users@qpid.apache.org; Donohue, Matt; brian@fluggo.com
Subject: Re: QPID message throughput - Red Hat numbers
A couple of interesting comments from the red hat doc.
The Intel® Xeon® 5482 based system increases throughput by ~ 48% over the Intel® Xeon® 5365 based system. [On average, 344K messages/sec for the Intel® Xeon® 5365 based system versus 505K messages/sec for the Intel® Xeon® 5482 based system.]
The optimized memory allocator increased throughput by ~ 49.7%. [On average, 558 messages/sec for the Intel® Xeon® 5365 based system versus 762K messages/sec for the Intel® Xeon® 5482 based system. Intel® Xeon® 5482 based system increased throughput by 36.6%.]
--- On Mon, 7/26/10, Ian.Kinkade <ki...@idi-middleware.com> wrote:
> From: Ian.Kinkade <ki...@idi-middleware.com>
> Subject: Re: QPID message throughput - Red Hat numbers
> To: users@qpid.apache.org, mdonohue@structure-tech.com, brian@fluggo.com
> Date: Monday, July 26, 2010, 6:30 PM
> Hi Matt & Brian,
>
> It is my understanding that the Red Hat tests were
> conducted using a
> Real-time Version of RHEL (MRG) and that it was
> specifically tuned for
> MRG-M and its test applications.
>
> You might want to try using the tuning application from the
> MRG install
> before you run the tests.
>
> I hope this was helpful?
>
> Best Regards .................... Ian
>
> Ian Kinkade
> CEO
> Information Design, Inc.
> 145 Durham Road, Suite 11
> Madison, CT 06443 USA
> URL: www.idi-middleware.com
> Email: kinkade@idi-middleware.com
>
> Work: 203-245-0772 Ext: 6212
> Fax: 203-245-1885
> Cell: 203-589-1192
>
>
> On 7/26/2010 7:54 PM, Donohue, Matt wrote:
> > The last project I worked on was the same for me. Not
> close to the MRG throughput numbers with the same test and
> this was on an otherwise optimized trading box.
> > The MRG qpid rpm was faster than an Intel C++ compiled
> version though.
> >
> > Regards,
> > Matt
> >
> > -----Original Message-----
> > From: Brian Crowell [mailto:brian@fluggo.com]
> > Sent: Monday, July 26, 2010 3:18 PM
> > To: users@qpid.apache.org
> > Subject: QPID message throughput - Red Hat numbers
> >
> > Red Hat claims to be able to get hundreds of thousands
> of messages
> > through on an eight core machine
> > (http://www.redhat.com/mrg/messaging/features/ or
> > http://www.redhat.com/f/pdf/mrg/Reference_Architecture_MRG_Messaging_Throughput.pdf).
> > I'm working with an eight-core machine, and I'm only
> getting about
> > 11,000/sec (in; about 6,500/sec out). This is with
> perftest, default
> > settings.
> >
> > What kinds of things do I need to be doing to get
> better throughput?
> >
> > --Brian
> >
> >
> ---------------------------------------------------------------------
> > Apache Qpid - AMQP Messaging Implementation
> > Project: http://qpid.apache.org
> > Use/Interact: mailto:users-subscribe@qpid.apache.org
> >
> >
> >
> ---------------------------------------------------------------------
> > Apache Qpid - AMQP Messaging Implementation
> > Project: http://qpid.apache.org
> > Use/Interact: mailto:users-subscribe@qpid.apache.org
> >
> >
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>
>
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: QPID message throughput - Red Hat numbers
Posted by Clark O'Brien <cl...@yahoo.com>.
A couple of interesting comments from the red hat doc.
The Intel® Xeon® 5482 based system increases throughput by ~ 48% over the Intel® Xeon® 5365 based system. [On average, 344K messages/sec for the Intel® Xeon® 5365 based system versus 505K messages/sec for the Intel® Xeon® 5482 based system.]
The optimized memory allocator increased throughput by ~ 49.7%. [On average, 558 messages/sec for the Intel® Xeon® 5365 based system versus 762K messages/sec for the Intel® Xeon® 5482 based system. Intel® Xeon® 5482 based system increased throughput by 36.6%.]
--- On Mon, 7/26/10, Ian.Kinkade <ki...@idi-middleware.com> wrote:
> From: Ian.Kinkade <ki...@idi-middleware.com>
> Subject: Re: QPID message throughput - Red Hat numbers
> To: users@qpid.apache.org, mdonohue@structure-tech.com, brian@fluggo.com
> Date: Monday, July 26, 2010, 6:30 PM
> Hi Matt & Brian,
>
> It is my understanding that the Red Hat tests were
> conducted using a
> Real-time Version of RHEL (MRG) and that it was
> specifically tuned for
> MRG-M and its test applications.
>
> You might want to try using the tuning application from the
> MRG install
> before you run the tests.
>
> I hope this was helpful?
>
> Best Regards .................... Ian
>
> Ian Kinkade
> CEO
> Information Design, Inc.
> 145 Durham Road, Suite 11
> Madison, CT 06443 USA
> URL: www.idi-middleware.com
> Email: kinkade@idi-middleware.com
>
> Work: 203-245-0772 Ext: 6212
> Fax: 203-245-1885
> Cell: 203-589-1192
>
>
> On 7/26/2010 7:54 PM, Donohue, Matt wrote:
> > The last project I worked on was the same for me. Not
> close to the MRG throughput numbers with the same test and
> this was on an otherwise optimized trading box.
> > The MRG qpid rpm was faster than an Intel C++ compiled
> version though.
> >
> > Regards,
> > Matt
> >
> > -----Original Message-----
> > From: Brian Crowell [mailto:brian@fluggo.com]
> > Sent: Monday, July 26, 2010 3:18 PM
> > To: users@qpid.apache.org
> > Subject: QPID message throughput - Red Hat numbers
> >
> > Red Hat claims to be able to get hundreds of thousands
> of messages
> > through on an eight core machine
> > (http://www.redhat.com/mrg/messaging/features/ or
> > http://www.redhat.com/f/pdf/mrg/Reference_Architecture_MRG_Messaging_Throughput.pdf).
> > I'm working with an eight-core machine, and I'm only
> getting about
> > 11,000/sec (in; about 6,500/sec out). This is with
> perftest, default
> > settings.
> >
> > What kinds of things do I need to be doing to get
> better throughput?
> >
> > --Brian
> >
> >
> ---------------------------------------------------------------------
> > Apache Qpid - AMQP Messaging Implementation
> > Project: http://qpid.apache.org
> > Use/Interact: mailto:users-subscribe@qpid.apache.org
> >
> >
> >
> ---------------------------------------------------------------------
> > Apache Qpid - AMQP Messaging Implementation
> > Project: http://qpid.apache.org
> > Use/Interact: mailto:users-subscribe@qpid.apache.org
> >
> >
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>
>
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: QPID message throughput - Red Hat numbers
Posted by "Ian.Kinkade" <ki...@idi-middleware.com>.
Hi Matt & Brian,
It is my understanding that the Red Hat tests were conducted using a
Real-time Version of RHEL (MRG) and that it was specifically tuned for
MRG-M and its test applications.
You might want to try using the tuning application from the MRG install
before you run the tests.
I hope this was helpful?
Best Regards .................... Ian
Ian Kinkade
CEO
Information Design, Inc.
145 Durham Road, Suite 11
Madison, CT 06443 USA
URL: www.idi-middleware.com
Email: kinkade@idi-middleware.com
Work: 203-245-0772 Ext: 6212
Fax: 203-245-1885
Cell: 203-589-1192
On 7/26/2010 7:54 PM, Donohue, Matt wrote:
> The last project I worked on was the same for me. Not close to the MRG throughput numbers with the same test and this was on an otherwise optimized trading box.
> The MRG qpid rpm was faster than an Intel C++ compiled version though.
>
> Regards,
> Matt
>
> -----Original Message-----
> From: Brian Crowell [mailto:brian@fluggo.com]
> Sent: Monday, July 26, 2010 3:18 PM
> To: users@qpid.apache.org
> Subject: QPID message throughput - Red Hat numbers
>
> Red Hat claims to be able to get hundreds of thousands of messages
> through on an eight core machine
> (http://www.redhat.com/mrg/messaging/features/ or
> http://www.redhat.com/f/pdf/mrg/Reference_Architecture_MRG_Messaging_Throughput.pdf).
> I'm working with an eight-core machine, and I'm only getting about
> 11,000/sec (in; about 6,500/sec out). This is with perftest, default
> settings.
>
> What kinds of things do I need to be doing to get better throughput?
>
> --Brian
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>
>
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
RE: QPID message throughput - Red Hat numbers
Posted by "Donohue, Matt" <md...@structure-tech.com>.
The last project I worked on was the same for me. Not close to the MRG throughput numbers with the same test and this was on an otherwise optimized trading box.
The MRG qpid rpm was faster than an Intel C++ compiled version though.
Regards,
Matt
-----Original Message-----
From: Brian Crowell [mailto:brian@fluggo.com]
Sent: Monday, July 26, 2010 3:18 PM
To: users@qpid.apache.org
Subject: QPID message throughput - Red Hat numbers
Red Hat claims to be able to get hundreds of thousands of messages
through on an eight core machine
(http://www.redhat.com/mrg/messaging/features/ or
http://www.redhat.com/f/pdf/mrg/Reference_Architecture_MRG_Messaging_Throughput.pdf).
I'm working with an eight-core machine, and I'm only getting about
11,000/sec (in; about 6,500/sec out). This is with perftest, default
settings.
What kinds of things do I need to be doing to get better throughput?
--Brian
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org