You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@qpid.apache.org by Kim van der Riet <ki...@redhat.com> on 2007/02/19 20:21:07 UTC
ApacheCon Asia 2006 - performance figures for Qpid
The following is from the Mina documentation page - it is a presentation
given at ApacheCon Asia 2006. On page 7 is a performance benchmark -
based on AMQP (and presumably Qpid!)
http://mina.apache.org/documentation.data/ACAsia2006.pdf
<snip>
Advantages: Performance
• AMQP Test
• Client and server
• 4 dual-core Opterons
• Via Gigabits Ethernet
• 10 clients
• Payload: 256+ bytes (excl. AMQP headers)
• Avg : 180,000 msg/sec
• Max: 220,000 msg/sec
Courtesy of Robert J. Greig, JP Morgan Chase & Co.
</snip>
Kim
Re: ApacheCon Asia 2006 - performance figures for Qpid
Posted by John O'Hara <jo...@gmail.com>.
You said
"You don't have to use VS to do in-memory replication. The primary can send
messages to the backup and when the transaction commits it can get an ack
from the backup. Then the transaction can be forced to disk later."
I like that model. People like solutions they 'think' they understand, and
that is something people think they understand...
Thanks for the book reference - I've got some reading to do!
John
On 22/02/07, lichtner@bway.net <li...@bway.net> wrote:
>
> > I've always been slightly wary of the tight/intimate type of cluster
> that
> > VS
> > solutions end up giving. I'm more of a fan of the first part of your
> > reply:
> >
> > If you replicate the messages in memory (to another machine or two) you
> do
> >> not have to force to disk upon commit - you can thus take the disk out
> >> of
> >> the critical path. Therefore you can get great throughput even with
> >> short
> >> transactions and short messages.
> >>
> >
> >
> > Two server solutions usually seem more robust than 3+ server solutions;
> > just
> > my experience of the Real World. Usually people don't invest enough on
> > the
> > infrastructure to make N-way solutions fail deterministically (e.g. they
> > skimp on network connections and multi-pathing).
>
> You don't have to use VS to do in-memory replication. The primary can send
> messages to the backup and when the transaction commits it can get an ack
> from the backup. Then the transaction can be forced to disk later.
>
> But the reasons I would use the totem protocol in this application are:
>
> 1) Using totem it's easy to do active/active, which seems more desirable
> than primary-backup. Deadlocks are not a problem because all servers see
> the same messages, and all messages arrive with a globally unique id.
>
> 2) There is a complex relationship between the degree of replication and
> availability, and the optimal availability is probably more than 2
> servers. It's interesting what you say about network availability, but VS
> is based on the concept that nodes can drop off the group at any time, so
> network availability need not be perfect.
>
> 3) Totem provides a really easy programming model, which makes the code
> more likely to be correct early on, which itself has an impact on
> availability.
>
> > I am interested in how we could add VS to Qpid/Java though. Could you
> > point
> > us to a "Dummy's Guide" for this application of VS (obviously using
> EVS4J
> > :-) We don't have the brainwidth to be expert in VS in addition to
> > working
> > with AMQP and Qpid.
>
> You could read the section called "Advanced Replication Techniques", in
> Ken Birman's book "Building Secure and Reliable Network Applications."
> It's available for free on the web. It references some other sections so
> you may have to go back and look at those. Birman uses causal ordering,
> not total ordering. And of course you can the read the totem article
> referenced on the evs4j web page.
>
> With EVS4J what you need to know is you get an api that allows you to send
> a message. Each member gets the message, including the sender, and when
> the message arrives it arrives in the same order at each member. When a
> fault occurs, you get notified of the membership change. And you get
> pretty good throughput.
>
>
>
Re: ApacheCon Asia 2006 - performance figures for Qpid
Posted by li...@bway.net.
> I've always been slightly wary of the tight/intimate type of cluster that
> VS
> solutions end up giving. I'm more of a fan of the first part of your
> reply:
>
> If you replicate the messages in memory (to another machine or two) you do
>> not have to force to disk upon commit - you can thus take the disk out
>> of
>> the critical path. Therefore you can get great throughput even with
>> short
>> transactions and short messages.
>>
>
>
> Two server solutions usually seem more robust than 3+ server solutions;
> just
> my experience of the Real World. Usually people don't invest enough on
> the
> infrastructure to make N-way solutions fail deterministically (e.g. they
> skimp on network connections and multi-pathing).
You don't have to use VS to do in-memory replication. The primary can send
messages to the backup and when the transaction commits it can get an ack
from the backup. Then the transaction can be forced to disk later.
But the reasons I would use the totem protocol in this application are:
1) Using totem it's easy to do active/active, which seems more desirable
than primary-backup. Deadlocks are not a problem because all servers see
the same messages, and all messages arrive with a globally unique id.
2) There is a complex relationship between the degree of replication and
availability, and the optimal availability is probably more than 2
servers. It's interesting what you say about network availability, but VS
is based on the concept that nodes can drop off the group at any time, so
network availability need not be perfect.
3) Totem provides a really easy programming model, which makes the code
more likely to be correct early on, which itself has an impact on
availability.
> I am interested in how we could add VS to Qpid/Java though. Could you
> point
> us to a "Dummy's Guide" for this application of VS (obviously using EVS4J
> :-) We don't have the brainwidth to be expert in VS in addition to
> working
> with AMQP and Qpid.
You could read the section called "Advanced Replication Techniques", in
Ken Birman's book "Building Secure and Reliable Network Applications."
It's available for free on the web. It references some other sections so
you may have to go back and look at those. Birman uses causal ordering,
not total ordering. And of course you can the read the totem article
referenced on the evs4j web page.
With EVS4J what you need to know is you get an api that allows you to send
a message. Each member gets the message, including the sender, and when
the message arrives it arrives in the same order at each member. When a
fault occurs, you get notified of the membership change. And you get
pretty good throughput.
Re: ApacheCon Asia 2006 - performance figures for Qpid
Posted by John O'Hara <jo...@gmail.com>.
You seem to know rather too much about this, so I Googled you :-)
and found out you wrote EVS4J - Kudos!
I've always been slightly wary of the tight/intimate type of cluster that VS
solutions end up giving. I'm more of a fan of the first part of your reply:
If you replicate the messages in memory (to another machine or two) you do
> not have to force to disk upon commit - you can thus take the disk out of
> the critical path. Therefore you can get great throughput even with short
> transactions and short messages.
>
Two server solutions usually seem more robust than 3+ server solutions; just
my experience of the Real World. Usually people don't invest enough on the
infrastructure to make N-way solutions fail deterministically (e.g. they
skimp on network connections and multi-pathing).
I am interested in how we could add VS to Qpid/Java though. Could you point
us to a "Dummy's Guide" for this application of VS (obviously using EVS4J
:-) We don't have the brainwidth to be expert in VS in addition to working
with AMQP and Qpid.
Many thanks for sharing your wisdom with us.
John
Re: ApacheCon Asia 2006 - performance figures for Qpid
Posted by li...@bway.net.
> You see greater performance if you commit in batches rather than after
> each
> message, as this means fewer disk synchs... Similarly if the message size
> is larger then you get greater Kb/s throughput.
If you replicate the messages in memory (to another machine or two) you do
not have to force to disk upon commit - you can thus take the disk out of
the critical path. Therefore you can get great throughput even with short
transactions and short messages.
With the totem protocol you can create, say, 3 copies of all messages with
a few milliseconds latency and a throughput of 25,000 messages per second
that's with 1423-byte messages.
Re: ApacheCon Asia 2006 - performance figures for Qpid
Posted by Robert Godfrey <ro...@gmail.com>.
OK, attached (Open Office Calc format) is the set of performance number that
I did over the weekend. These numbers are from NAS storage, the faster
number I mentioned above (1100 Msg/s) is from SAN - I haven't yet had time
to retest against the SAN but I think these numbers are interesting in
themselves.
To explain the numbers a little, each line represents a test where a number
(given in the Queues column) of Sender/receiver pairs are started, each on a
separate queue. Each pair sends/receives 1000 messages (so if there are 8
queues then 8000 messages are sent). The Msg/s and Kb/s are totals across
all pairs.
You see greater performance if you commit in batches rather than after each
message, as this means fewer disk synchs... Similarly if the message size
is larger then you get greater Kb/s throughput.
These numbers are Java Client -> Jave Broker -> Java Client, using the BDB
store. Would be interesting to try some other combinations.
-- Rob
On 20/02/07, John O'Hara <jo...@gmail.com> wrote:
>
> Not at all shabby.
>
> On 19/02/07, Robert Godfrey <ro...@gmail.com> wrote:
> >
> > The current 0-8 trunk can do slightly better that that now
> :-) Certainly
> > seen peek over 250,000 msg/s.
> >
> > Recently I've been testing persistent messages on the same hardware,
> using
> > point-to-point messaging (direct exchange)
> >
> > With 8 producers, and 8 receivers (all located on the same machine),
> 1Kb
> > message size, and each producer-receiver pair using a separate queue; I
> > see
> > about 1100 round-trip messages per second (that its 2200 total message
> > transfers). This is using transactions, with a commit after each
> publish
> > or
> > deliver. Better performance can be had by using larger commit batches.
> >
> > When I get time, I'll tabulate the results.
> >
> >
> > -- Rob
> >
> > On 19/02/07, Kim van der Riet <ki...@redhat.com> wrote:
> > >
> > > The following is from the Mina documentation page - it is a
> presentation
> > > given at ApacheCon Asia 2006. On page 7 is a performance benchmark -
> > > based on AMQP (and presumably Qpid!)
> > >
> > > http://mina.apache.org/documentation.data/ACAsia2006.pdf
> > >
> > > <snip>
> > > Advantages: Performance
> > > • AMQP Test
> > > • Client and server
> > > • 4 dual-core Opterons
> > > • Via Gigabits Ethernet
> > > • 10 clients
> > > • Payload: 256+ bytes (excl. AMQP headers)
> > > • Avg : 180,000 msg/sec
> > > • Max: 220,000 msg/sec
> > > Courtesy of Robert J. Greig, JP Morgan Chase & Co.
> > > </snip>
> > >
> > > Kim
> > >
> > >
> >
>
Re: ApacheCon Asia 2006 - performance figures for Qpid
Posted by John O'Hara <jo...@gmail.com>.
Not at all shabby.
On 19/02/07, Robert Godfrey <ro...@gmail.com> wrote:
>
> The current 0-8 trunk can do slightly better that that now :-) Certainly
> seen peek over 250,000 msg/s.
>
> Recently I've been testing persistent messages on the same hardware, using
> point-to-point messaging (direct exchange)
>
> With 8 producers, and 8 receivers (all located on the same machine), 1Kb
> message size, and each producer-receiver pair using a separate queue; I
> see
> about 1100 round-trip messages per second (that its 2200 total message
> transfers). This is using transactions, with a commit after each publish
> or
> deliver. Better performance can be had by using larger commit batches.
>
> When I get time, I'll tabulate the results.
>
>
> -- Rob
>
> On 19/02/07, Kim van der Riet <ki...@redhat.com> wrote:
> >
> > The following is from the Mina documentation page - it is a presentation
> > given at ApacheCon Asia 2006. On page 7 is a performance benchmark -
> > based on AMQP (and presumably Qpid!)
> >
> > http://mina.apache.org/documentation.data/ACAsia2006.pdf
> >
> > <snip>
> > Advantages: Performance
> > • AMQP Test
> > • Client and server
> > • 4 dual-core Opterons
> > • Via Gigabits Ethernet
> > • 10 clients
> > • Payload: 256+ bytes (excl. AMQP headers)
> > • Avg : 180,000 msg/sec
> > • Max: 220,000 msg/sec
> > Courtesy of Robert J. Greig, JP Morgan Chase & Co.
> > </snip>
> >
> > Kim
> >
> >
>
Re: ApacheCon Asia 2006 - performance figures for Qpid
Posted by Robert Godfrey <ro...@gmail.com>.
The current 0-8 trunk can do slightly better that that now :-) Certainly
seen peek over 250,000 msg/s.
Recently I've been testing persistent messages on the same hardware, using
point-to-point messaging (direct exchange)
With 8 producers, and 8 receivers (all located on the same machine), 1Kb
message size, and each producer-receiver pair using a separate queue; I see
about 1100 round-trip messages per second (that its 2200 total message
transfers). This is using transactions, with a commit after each publish or
deliver. Better performance can be had by using larger commit batches.
When I get time, I'll tabulate the results.
-- Rob
On 19/02/07, Kim van der Riet <ki...@redhat.com> wrote:
>
> The following is from the Mina documentation page - it is a presentation
> given at ApacheCon Asia 2006. On page 7 is a performance benchmark -
> based on AMQP (and presumably Qpid!)
>
> http://mina.apache.org/documentation.data/ACAsia2006.pdf
>
> <snip>
> Advantages: Performance
> • AMQP Test
> • Client and server
> • 4 dual-core Opterons
> • Via Gigabits Ethernet
> • 10 clients
> • Payload: 256+ bytes (excl. AMQP headers)
> • Avg : 180,000 msg/sec
> • Max: 220,000 msg/sec
> Courtesy of Robert J. Greig, JP Morgan Chase & Co.
> </snip>
>
> Kim
>
>