You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by Jason Barto <ja...@gmail.com> on 2013/03/25 19:20:34 UTC
Performance degradation over time during amqp 1.0 stress test
Using amqp 0.9.1 I'm getting an average performance of 70k messages per
second throughput during an hour long test.
When using amqp 1.0 I receive similar but smaller throughput numbers
however after about 90 seconds of testing this quickly begins to drop until
it gets to 10s or 100s of messages per second and a significant backlog of
messages begins building.
Can anyone explain this drastic difference in performance? I am using v0.20
of qpid-j for the brokering.
Sincerely,
Jason
Re: Performance degradation over time during amqp 1.0 stress test
Posted by Jason Barto <ja...@gmail.com>.
Rob, will do. I'd also like to try taking a look at the source and see if I
can identify the cause of the seeming memory leak. Although my java is so
rusty I doubt I'll find anything.
Regarding your question about the protocol planned for the 20k messages, we
have ever changing requirements so while I'm happy to build my solution on
0.9.1 I'd like to future proof by helping qpid develop to support the same
incredible throughput in 1.0 that we see in 0.9.1.
Sincerely,
Jason
On Wednesday, March 27, 2013, Rob Godfrey wrote:
> Hi Jason,
>
> OK... so that's embarrassing :-) The reason that the broker stopped sending
> out messages in my test case was due to a deadlock in the broker side AMQP
> 1.0 codepath [1]. I've applied a fix to the trunk version of the broker,
> which I'll presently be asking for inclusion into the 0.22 release.
>
> Can you see if this also fixes the issue you have been seeing?
>
> Cheers,
> Rob
>
> [1] http://svn.apache.org/viewvc?view=revision&revision=r1461844
>
> On 27 March 2013 20:41, Rob Godfrey <ro...@gmail.com> wrote:
>
> > Hi Jason,
> >
> > just to let you know I am investigating the issue where the broker
> > suddenly stops receiving messages... I can see this too but I haven't yet
> > got to the bottom of why. Apologies for any inconvenience.
> >
> > Cheers,
> > Rob
> >
> >
> > On 27 March 2013 17:51, Jason Barto <ja...@gmail.com> wrote:
> >
> >> Rob,
> >> first let me say thank you so much for your insight and quick responses.
> >> I'm sorry to say that throttling the producers has not completely solved
> >> the problem. I'm back in the office today and have been doing
> additional
> >> testing and I think I may have detected a memory leak in the code that
> >> handles AMQP 1.0 communications.
> >>
> >> My experience thus far:
> >> I downloaded and installed the latest QPid revision using the git
> >> directions on the website. With the Java QPid up and running I ran my
> >> first test client (AMQP 0-9-1). I sent 60k messages/sec, each with a
> 1KB
> >> payload, for 180 seconds. Using top I monitored the JVMs memory usage
> >> which held steady at 233MB. The code for this client can be found on
> >> pastebin at http://pastebin.com/fwDmhxG9. It uses the rabbitmq-client
> >> JAR
> >> for AMQP 0-9-1.
> >>
> >> Again using 'top' and the latest Java QPid I ran my second test client
> at
> >> 8k messages/sec, each with 1 KB payload, for 180 seconds. The memory
> >> utilization of the JVM reported by 'top' rose steadily until the test
> >> ended, at which time the JVM was using 1.1GB of memory. The code for
> this
> >> second test client is at http://pastebin.com/DNTLCj0y and uses v0.23 of
> >> the
> >> qpid-amqp-1-0-client JAR. It may also be worth noting that at about
> >> second
> >> 147 the consumer just stopped receiving messages, no exceptions were
> >> thrown
> >> on either the client or broker, it simply stopped receiving messages.
> >>
> >> Both tests were throttled to ensure that the consumer would consume in a
> >> timely fashion roughly all messages sent, to avoid any buildup of
> messages
> >> in the broker's queue.
> >>
> >> I will admit that I have far more experience with the RabbitMQ client
> >> library than the QPid library so my hope is that there is something
> >> someone
> >> can spot in my test code that is not being done correctly that will
> >> explain
> >> why the JVM is consuming so much memory. Does acknowledging a message
> >> with
> >> Receiver.acknowledge not sufficiently inform the broker that the message
> >> has been received and can be forgotten?
> >>
> >> On longer test runs, 500+ seconds, the broker eventually consumed all of
> >> its allocated ram and as you mentioned the GC began taking over trying
> to
> >> keep things running.
> >>
> >> Sincerely,
> >> Jason
> >>
> >>
> >> On Tue, Mar 26, 2013 at 12:43 PM, Rob Godfrey <rob.j.godfrey@gmail.com
> >> >wrote:
> >>
> >> > Hi Jason,
> >> >
> >> > On 26 March 2013 13:26, Jason Barto <ja...@gmail.com> wrote:
> >> >
> >> > > Rob,
> >> > > thanks for your quick response - I've consolidated the code into a
> >> single
> >> > > java file and would gladly publish it - frankly being new to AMQP
> 1.0
> >> and
> >> > > its client libraries I'd value the feedback. I think I may have
> >> > determined
> >> > > the reason for the slow down - not sure why it didn't occur to me
> >> earlier
> >> > > to be honest.
> >> > >
> >> > > I have 2 threads, a Sender and Receiver. Both are trying to
> produce /
> >> > > consume as rapidly as possible. Testing on my laptop the Sender is
> >> > > publishing on average 20k messages / sec while the receiver seems to
> >> top
> >>
Re: Performance degradation over time during amqp 1.0 stress test
Posted by Robbie Gemmell <ro...@gmail.com>.
Can I ask what virtualhost message store configuation you were using for
the two brokers?
I ask as 0.20 shipped with an example virtualhost config which uses the
Memory message store, as all prior releases have, but trunk (and 0.22 once
released) has a default virtualhost configured with the Derby message
store. This would lead to a considerable difference in performance if
persistent messages are being sent simply because the two brokers would
actually be doing different things, with one keeping all the messages in
memory and the other writing them to disk. That might also look like a
reduction in memory use when in reality it would just be a reduced rate of
usage.
Robbie
On 6 April 2013 00:46, Jason Barto <ja...@gmail.com> wrote:
> Rob,
> I've had a chance to review the latest updates to QPid. And while the
> memory leak has been lessened it has not been removed (to be expected I
> think). However, on the downsie, the performance of the AMQP 1.0 code
> seems to have declined significantly. I'm attaching the results from 2
> performance tests. In each a single consumer and single producer is
> communicating with the QPid-J broker, using messages of roughly 1KB in
> size. (I say roughly because it is a timestamp plus garbage data). The 2
> brokers compared were Qpid-J v0.20 and the latest pull from the GIT
> repository. Both tests were stopped after the host system (in this case
> Macbook Pro, Intel Core i7, 64bit, 4 core, 8 GB ram) reported (via top)
> that the JVM had consumed resident private address space exceeding 900 MB
> with an mx of 1024.
>
> The attachments show the number of messages sent and recieved per second,
> followed by the latency of the messages. Where I'm concerned is in the
> throughput - QPid 0.20 demonstrates an AMQP 1.0 throughput n the order of
> 12K messages per second but with a large leak into ram. Your recent work
> into optimizing the 1.0 Java code has significantly reduced this, allowing
> the tests to run for longer, however the performance is greatly reduced to
> about 2000 messages per second.
>
> Looking at VisualVM on OSX (I'm new to VisualVM), in the profiler for the
> QPid proces, shows that the largest class allocation in the QPid proces, in
> both cases, is of type 'byte[]'. Not having dug into the code yet I'm not
> sure what necessarily this means or even if its of any help. However I'm
> attaching my test results in addition. I'll also say that while I've
> gotten similar results using both client code bases, the tests attached to
> this email were collected using the SwiftMQ Java client library.
>
> Sincerely,
> Jason
>
>
> On Wed, Mar 27, 2013 at 8:45 PM, Rob Godfrey <ro...@gmail.com>wrote:
>
>> Hi Jason,
>>
>> OK... so that's embarrassing :-) The reason that the broker stopped
>> sending
>> out messages in my test case was due to a deadlock in the broker side AMQP
>> 1.0 codepath [1]. I've applied a fix to the trunk version of the broker,
>> which I'll presently be asking for inclusion into the 0.22 release.
>>
>> Can you see if this also fixes the issue you have been seeing?
>>
>> Cheers,
>> Rob
>>
>> [1] http://svn.apache.org/viewvc?view=revision&revision=r1461844
>>
>> On 27 March 2013 20:41, Rob Godfrey <ro...@gmail.com> wrote:
>>
>> > Hi Jason,
>> >
>> > just to let you know I am investigating the issue where the broker
>> > suddenly stops receiving messages... I can see this too but I haven't
>> yet
>> > got to the bottom of why. Apologies for any inconvenience.
>> >
>> > Cheers,
>> > Rob
>> >
>> >
>> > On 27 March 2013 17:51, Jason Barto <ja...@gmail.com> wrote:
>> >
>> >> Rob,
>> >> first let me say thank you so much for your insight and quick
>> responses.
>> >> I'm sorry to say that throttling the producers has not completely
>> solved
>> >> the problem. I'm back in the office today and have been doing
>> additional
>> >> testing and I think I may have detected a memory leak in the code that
>> >> handles AMQP 1.0 communications.
>> >>
>> >> My experience thus far:
>> >> I downloaded and installed the latest QPid revision using the git
>> >> directions on the website. With the Java QPid up and running I ran my
>> >> first test client (AMQP 0-9-1). I sent 60k messages/sec, each with a
>> 1KB
>> >> payload, for 180 seconds. Using top I monitored the JVMs memory usage
>> >> which held steady at 233MB. The code for this client can be found on
>> >> pastebin at http://pastebin.com/fwDmhxG9. It uses the rabbitmq-client
>> >> JAR
>> >> for AMQP 0-9-1.
>> >>
>> >> Again using 'top' and the latest Java QPid I ran my second test client
>> at
>> >> 8k messages/sec, each with 1 KB payload, for 180 seconds. The memory
>> >> utilization of the JVM reported by 'top' rose steadily until the test
>> >> ended, at which time the JVM was using 1.1GB of memory. The code for
>> this
>> >> second test client is at http://pastebin.com/DNTLCj0y and uses v0.23
>> of
>> >> the
>> >> qpid-amqp-1-0-client JAR. It may also be worth noting that at about
>> >> second
>> >> 147 the consumer just stopped receiving messages, no exceptions were
>> >> thrown
>> >> on either the client or broker, it simply stopped receiving messages.
>> >>
>> >> Both tests were throttled to ensure that the consumer would consume in
>> a
>> >> timely fashion roughly all messages sent, to avoid any buildup of
>> messages
>> >> in the broker's queue.
>> >>
>> >> I will admit that I have far more experience with the RabbitMQ client
>> >> library than the QPid library so my hope is that there is something
>> >> someone
>> >> can spot in my test code that is not being done correctly that will
>> >> explain
>> >> why the JVM is consuming so much memory. Does acknowledging a message
>> >> with
>> >> Receiver.acknowledge not sufficiently inform the broker that the
>> message
>> >> has been received and can be forgotten?
>> >>
>> >> On longer test runs, 500+ seconds, the broker eventually consumed all
>> of
>> >> its allocated ram and as you mentioned the GC began taking over trying
>> to
>> >> keep things running.
>> >>
>> >> Sincerely,
>> >> Jason
>> >>
>> >>
>> >> On Tue, Mar 26, 2013 at 12:43 PM, Rob Godfrey <rob.j.godfrey@gmail.com
>> >> >wrote:
>> >>
>> >> > Hi Jason,
>> >> >
>> >> > On 26 March 2013 13:26, Jason Barto <ja...@gmail.com> wrote:
>> >> >
>> >> > > Rob,
>> >> > > thanks for your quick response - I've consolidated the code into a
>> >> single
>> >> > > java file and would gladly publish it - frankly being new to AMQP
>> 1.0
>> >> and
>> >> > > its client libraries I'd value the feedback. I think I may have
>> >> > determined
>> >> > > the reason for the slow down - not sure why it didn't occur to me
>> >> earlier
>> >> > > to be honest.
>> >> > >
>> >> > > I have 2 threads, a Sender and Receiver. Both are trying to
>> produce /
>> >> > > consume as rapidly as possible. Testing on my laptop the Sender is
>> >> > > publishing on average 20k messages / sec while the receiver seems
>> to
>> >> top
>> >> > > out at 15K msg/sec, leading to an inevitable backlog of messages
>> >> building
>> >> > > up on the broker. When the queue on the broker is holding around
>> 500k
>> >> > > messages (after about 90 seconds of testing) the performance of the
>> >> > broker
>> >> > > drops dramatically until both producer / consumer can only send /
>> >> receive
>> >> > > about 200 msg/s.
>> >> > >
>> >> > >
>> >> >
>> >> > OK - great (sort of) - we can understand the problem - basically the
>> >> broker
>> >> > is not sufficiently pushing back on the producing client when the
>> queue
>> >> > becomes over full.
>> >> >
>> >> > The broker can be used to enforce flow control when using earlier
>> >> versions
>> >> > of the protocol (0-8, 0-9, 0-9-1, 0-10) but it's entirely possible
>> that
>> >> > this is not yet implemented in the 1-0 codepath (I shall have to go
>> >> > check... if it is not I'll attempt to implement it shortly).
>> >> >
>> >> > In general the broker performance shouldn't degrade until it starts
>> >> running
>> >> > out of memory, whereupon the garbage collector will start dominating
>> the
>> >> > performance.
>> >> >
>> >> >
>> >> > > If I put a limit on the number of messages / sec sent by the
>> producer
>> >> the
>> >> > > problem goes away, ie the receiver can consume the messages at a
>> rate
>> >> to
>> >> > > prevent a backlog of messages. I'd be happy to create a JIRA
>> request
>> >> if
>> >> > > you feel one is necessary; is there perhaps a datastore I could
>> >> configure
>> >> > > QPid to use that would help the system to better deal with the
>> growing
>> >> > > backlog mentioned? I ask as a mitigation technique. I'll be
>> >> becoming a
>> >> > > sysadmin for an enterprise-level message broker that will be
>> expected
>> >> to
>> >> > > handle 1.75B messages per day (roughly 20k/s). If for some reason
>> the
>> >> > > messages are not being consumed at the same rate they're being
>> >> produced -
>> >> > > what options do I have to ensure that QPid's performance is
>> >> unaffected?
>> >> > >
>> >> > >
>> >> > I guess the question is how large do you want to let the queue grow
>> >> before
>> >> > you have to enforce some sort of restriction on the sender? Adding
>> >> > flow-to-disk style capabilities often just makes the problem worse
>> >> because
>> >> > the consumer starts to become limited by the speed at which messages
>> >> can be
>> >> > read from disk, which may be lower than it's optimal rate... so you
>> >> start
>> >> > seeing even more queue growth. I've seen people who want to be able
>> to
>> >> > handle spikes in queue growth of up to several million messages...
>> and
>> >> the
>> >> > answer to their problem was just to stuff a lot more RAM into their
>> >> machine
>> >> > :)
>> >> >
>> >> > As an aside are you expecting these 20K/s messages to be AMQP 1.0?
>> As I
>> >> > mentioned in my last mail, I've not yet done *any* perf tuning on the
>> >> > AMQP1.0 code (JMS client or broker)... Though your question yesterday
>> >> has
>> >> > forced me to dig up my old copy of JProfiler and run it on the broker
>> >> for
>> >> > the first time today :-)
>> >> >
>> >> > Cheers,
>> >> > Rob
>> >> >
>> >> >
>> >> > > Sincerely,
>> >> > > Jason
>> >> > >
>> >> > >
>> >> > > On Mon, Mar 25, 2013 at 7:08 PM, Rob Godfrey <
>> rob.j.godfrey@gmail.com
>> >> > > >wrote:
>> >> > >
>> >> > > > Hi Jason,
>> >> > > >
>> >> > > > the AMQP 1-0 support in the java broker is somewhat experimental
>> at
>> >> > > > this stage (and no perf tuning has been carried out), however it
>> >> > > > definitely seems like you've found a bug. Can you share some more
>> >> > > > details of your test (perhaps raise a JIRA and attach the test
>> >> code?)
>> >> > > > and I'll try to look into it ASAP - hopefully that way we can
>> get a
>> >> > > > fix in for 0.22,
>> >> > > >
>> >> > > > Thanks,
>> >> > > > Rob
>> >> > > >
>> >> > > > On 25 March 2013 19:20, Jason Barto <ja...@gmail.com>
>> >> wrote:
>> >> > > > > Using amqp 0.9.1 I'm getting an average performance of 70k
>> >> messages
>> >> > per
>> >> > > > > second throughput during an hour long test.
>> >> > > > >
>> >> > > > > When using amqp 1.0 I receive similar but smaller throughput
>> >> numbers
>> >> > > > > however after about 90 seconds of testing this quickly begins
>> to
>> >> drop
>> >> > > > until
>> >> > > > > it gets to 10s or 100s of messages per second and a significant
>> >> > backlog
>> >> > > > of
>> >> > > > > messages begins building.
>> >> > > > >
>> >> > > > > Can anyone explain this drastic difference in performance? I am
>> >> using
>> >> > > > v0.20
>> >> > > > > of qpid-j for the brokering.
>> >> > > > >
>> >> > > > > Sincerely,
>> >> > > > > Jason
>> >> > > >
>> >> > > >
>> >> ---------------------------------------------------------------------
>> >> > > > To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
>> >> > > > For additional commands, e-mail: users-help@qpid.apache.org
>> >> > > >
>> >> > > >
>> >> > >
>> >> >
>> >>
>> >
>> >
>>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
Re: Performance degradation over time during amqp 1.0 stress test
Posted by Rob Godfrey <ro...@gmail.com>.
Hi Jason,
thanks - I'll look into this. The changes to the broker to fix the
deadlock would have introduced more synchronization, but I wouldn't have
expected it to make a significant difference to performance. I've been a
bit snowed under with other stuff for the last few days - but I'll try to
look into this before the end of the week.
-- Rob
On 6 April 2013 01:46, Jason Barto <ja...@gmail.com> wrote:
> Rob,
> I've had a chance to review the latest updates to QPid. And while the
> memory leak has been lessened it has not been removed (to be expected I
> think). However, on the downsie, the performance of the AMQP 1.0 code
> seems to have declined significantly. I'm attaching the results from 2
> performance tests. In each a single consumer and single producer is
> communicating with the QPid-J broker, using messages of roughly 1KB in
> size. (I say roughly because it is a timestamp plus garbage data). The 2
> brokers compared were Qpid-J v0.20 and the latest pull from the GIT
> repository. Both tests were stopped after the host system (in this case
> Macbook Pro, Intel Core i7, 64bit, 4 core, 8 GB ram) reported (via top)
> that the JVM had consumed resident private address space exceeding 900 MB
> with an mx of 1024.
>
> The attachments show the number of messages sent and recieved per second,
> followed by the latency of the messages. Where I'm concerned is in the
> throughput - QPid 0.20 demonstrates an AMQP 1.0 throughput n the order of
> 12K messages per second but with a large leak into ram. Your recent work
> into optimizing the 1.0 Java code has significantly reduced this, allowing
> the tests to run for longer, however the performance is greatly reduced to
> about 2000 messages per second.
>
> Looking at VisualVM on OSX (I'm new to VisualVM), in the profiler for the
> QPid proces, shows that the largest class allocation in the QPid proces, in
> both cases, is of type 'byte[]'. Not having dug into the code yet I'm not
> sure what necessarily this means or even if its of any help. However I'm
> attaching my test results in addition. I'll also say that while I've
> gotten similar results using both client code bases, the tests attached to
> this email were collected using the SwiftMQ Java client library.
>
> Sincerely,
> Jason
>
>
> On Wed, Mar 27, 2013 at 8:45 PM, Rob Godfrey <ro...@gmail.com>wrote:
>
>> Hi Jason,
>>
>> OK... so that's embarrassing :-) The reason that the broker stopped
>> sending
>> out messages in my test case was due to a deadlock in the broker side AMQP
>> 1.0 codepath [1]. I've applied a fix to the trunk version of the broker,
>> which I'll presently be asking for inclusion into the 0.22 release.
>>
>> Can you see if this also fixes the issue you have been seeing?
>>
>> Cheers,
>> Rob
>>
>> [1] http://svn.apache.org/viewvc?view=revision&revision=r1461844
>>
>> On 27 March 2013 20:41, Rob Godfrey <ro...@gmail.com> wrote:
>>
>> > Hi Jason,
>> >
>> > just to let you know I am investigating the issue where the broker
>> > suddenly stops receiving messages... I can see this too but I haven't
>> yet
>> > got to the bottom of why. Apologies for any inconvenience.
>> >
>> > Cheers,
>> > Rob
>> >
>> >
>> > On 27 March 2013 17:51, Jason Barto <ja...@gmail.com> wrote:
>> >
>> >> Rob,
>> >> first let me say thank you so much for your insight and quick
>> responses.
>> >> I'm sorry to say that throttling the producers has not completely
>> solved
>> >> the problem. I'm back in the office today and have been doing
>> additional
>> >> testing and I think I may have detected a memory leak in the code that
>> >> handles AMQP 1.0 communications.
>> >>
>> >> My experience thus far:
>> >> I downloaded and installed the latest QPid revision using the git
>> >> directions on the website. With the Java QPid up and running I ran my
>> >> first test client (AMQP 0-9-1). I sent 60k messages/sec, each with a
>> 1KB
>> >> payload, for 180 seconds. Using top I monitored the JVMs memory usage
>> >> which held steady at 233MB. The code for this client can be found on
>> >> pastebin at http://pastebin.com/fwDmhxG9. It uses the rabbitmq-client
>> >> JAR
>> >> for AMQP 0-9-1.
>> >>
>> >> Again using 'top' and the latest Java QPid I ran my second test client
>> at
>> >> 8k messages/sec, each with 1 KB payload, for 180 seconds. The memory
>> >> utilization of the JVM reported by 'top' rose steadily until the test
>> >> ended, at which time the JVM was using 1.1GB of memory. The code for
>> this
>> >> second test client is at http://pastebin.com/DNTLCj0y and uses v0.23
>> of
>> >> the
>> >> qpid-amqp-1-0-client JAR. It may also be worth noting that at about
>> >> second
>> >> 147 the consumer just stopped receiving messages, no exceptions were
>> >> thrown
>> >> on either the client or broker, it simply stopped receiving messages.
>> >>
>> >> Both tests were throttled to ensure that the consumer would consume in
>> a
>> >> timely fashion roughly all messages sent, to avoid any buildup of
>> messages
>> >> in the broker's queue.
>> >>
>> >> I will admit that I have far more experience with the RabbitMQ client
>> >> library than the QPid library so my hope is that there is something
>> >> someone
>> >> can spot in my test code that is not being done correctly that will
>> >> explain
>> >> why the JVM is consuming so much memory. Does acknowledging a message
>> >> with
>> >> Receiver.acknowledge not sufficiently inform the broker that the
>> message
>> >> has been received and can be forgotten?
>> >>
>> >> On longer test runs, 500+ seconds, the broker eventually consumed all
>> of
>> >> its allocated ram and as you mentioned the GC began taking over trying
>> to
>> >> keep things running.
>> >>
>> >> Sincerely,
>> >> Jason
>> >>
>> >>
>> >> On Tue, Mar 26, 2013 at 12:43 PM, Rob Godfrey <rob.j.godfrey@gmail.com
>> >> >wrote:
>> >>
>> >> > Hi Jason,
>> >> >
>> >> > On 26 March 2013 13:26, Jason Barto <ja...@gmail.com> wrote:
>> >> >
>> >> > > Rob,
>> >> > > thanks for your quick response - I've consolidated the code into a
>> >> single
>> >> > > java file and would gladly publish it - frankly being new to AMQP
>> 1.0
>> >> and
>> >> > > its client libraries I'd value the feedback. I think I may have
>> >> > determined
>> >> > > the reason for the slow down - not sure why it didn't occur to me
>> >> earlier
>> >> > > to be honest.
>> >> > >
>> >> > > I have 2 threads, a Sender and Receiver. Both are trying to
>> produce /
>> >> > > consume as rapidly as possible. Testing on my laptop the Sender is
>> >> > > publishing on average 20k messages / sec while the receiver seems
>> to
>> >> top
>> >> > > out at 15K msg/sec, leading to an inevitable backlog of messages
>> >> building
>> >> > > up on the broker. When the queue on the broker is holding around
>> 500k
>> >> > > messages (after about 90 seconds of testing) the performance of the
>> >> > broker
>> >> > > drops dramatically until both producer / consumer can only send /
>> >> receive
>> >> > > about 200 msg/s.
>> >> > >
>> >> > >
>> >> >
>> >> > OK - great (sort of) - we can understand the problem - basically the
>> >> broker
>> >> > is not sufficiently pushing back on the producing client when the
>> queue
>> >> > becomes over full.
>> >> >
>> >> > The broker can be used to enforce flow control when using earlier
>> >> versions
>> >> > of the protocol (0-8, 0-9, 0-9-1, 0-10) but it's entirely possible
>> that
>> >> > this is not yet implemented in the 1-0 codepath (I shall have to go
>> >> > check... if it is not I'll attempt to implement it shortly).
>> >> >
>> >> > In general the broker performance shouldn't degrade until it starts
>> >> running
>> >> > out of memory, whereupon the garbage collector will start dominating
>> the
>> >> > performance.
>> >> >
>> >> >
>> >> > > If I put a limit on the number of messages / sec sent by the
>> producer
>> >> the
>> >> > > problem goes away, ie the receiver can consume the messages at a
>> rate
>> >> to
>> >> > > prevent a backlog of messages. I'd be happy to create a JIRA
>> request
>> >> if
>> >> > > you feel one is necessary; is there perhaps a datastore I could
>> >> configure
>> >> > > QPid to use that would help the system to better deal with the
>> growing
>> >> > > backlog mentioned? I ask as a mitigation technique. I'll be
>> >> becoming a
>> >> > > sysadmin for an enterprise-level message broker that will be
>> expected
>> >> to
>> >> > > handle 1.75B messages per day (roughly 20k/s). If for some reason
>> the
>> >> > > messages are not being consumed at the same rate they're being
>> >> produced -
>> >> > > what options do I have to ensure that QPid's performance is
>> >> unaffected?
>> >> > >
>> >> > >
>> >> > I guess the question is how large do you want to let the queue grow
>> >> before
>> >> > you have to enforce some sort of restriction on the sender? Adding
>> >> > flow-to-disk style capabilities often just makes the problem worse
>> >> because
>> >> > the consumer starts to become limited by the speed at which messages
>> >> can be
>> >> > read from disk, which may be lower than it's optimal rate... so you
>> >> start
>> >> > seeing even more queue growth. I've seen people who want to be able
>> to
>> >> > handle spikes in queue growth of up to several million messages...
>> and
>> >> the
>> >> > answer to their problem was just to stuff a lot more RAM into their
>> >> machine
>> >> > :)
>> >> >
>> >> > As an aside are you expecting these 20K/s messages to be AMQP 1.0?
>> As I
>> >> > mentioned in my last mail, I've not yet done *any* perf tuning on the
>> >> > AMQP1.0 code (JMS client or broker)... Though your question yesterday
>> >> has
>> >> > forced me to dig up my old copy of JProfiler and run it on the broker
>> >> for
>> >> > the first time today :-)
>> >> >
>> >> > Cheers,
>> >> > Rob
>> >> >
>> >> >
>> >> > > Sincerely,
>> >> > > Jason
>> >> > >
>> >> > >
>> >> > > On Mon, Mar 25, 2013 at 7:08 PM, Rob Godfrey <
>> rob.j.godfrey@gmail.com
>> >> > > >wrote:
>> >> > >
>> >> > > > Hi Jason,
>> >> > > >
>> >> > > > the AMQP 1-0 support in the java broker is somewhat experimental
>> at
>> >> > > > this stage (and no perf tuning has been carried out), however it
>> >> > > > definitely seems like you've found a bug. Can you share some more
>> >> > > > details of your test (perhaps raise a JIRA and attach the test
>> >> code?)
>> >> > > > and I'll try to look into it ASAP - hopefully that way we can
>> get a
>> >> > > > fix in for 0.22,
>> >> > > >
>> >> > > > Thanks,
>> >> > > > Rob
>> >> > > >
>> >> > > > On 25 March 2013 19:20, Jason Barto <ja...@gmail.com>
>> >> wrote:
>> >> > > > > Using amqp 0.9.1 I'm getting an average performance of 70k
>> >> messages
>> >> > per
>> >> > > > > second throughput during an hour long test.
>> >> > > > >
>> >> > > > > When using amqp 1.0 I receive similar but smaller throughput
>> >> numbers
>> >> > > > > however after about 90 seconds of testing this quickly begins
>> to
>> >> drop
>> >> > > > until
>> >> > > > > it gets to 10s or 100s of messages per second and a significant
>> >> > backlog
>> >> > > > of
>> >> > > > > messages begins building.
>> >> > > > >
>> >> > > > > Can anyone explain this drastic difference in performance? I am
>> >> using
>> >> > > > v0.20
>> >> > > > > of qpid-j for the brokering.
>> >> > > > >
>> >> > > > > Sincerely,
>> >> > > > > Jason
>> >> > > >
>> >> > > >
>> >> ---------------------------------------------------------------------
>> >> > > > To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
>> >> > > > For additional commands, e-mail: users-help@qpid.apache.org
>> >> > > >
>> >> > > >
>> >> > >
>> >> >
>> >>
>> >
>> >
>>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
Re: Performance degradation over time during amqp 1.0 stress test
Posted by Jason Barto <ja...@gmail.com>.
Rob,
I've had a chance to review the latest updates to QPid. And while the
memory leak has been lessened it has not been removed (to be expected I
think). However, on the downsie, the performance of the AMQP 1.0 code
seems to have declined significantly. I'm attaching the results from 2
performance tests. In each a single consumer and single producer is
communicating with the QPid-J broker, using messages of roughly 1KB in
size. (I say roughly because it is a timestamp plus garbage data). The 2
brokers compared were Qpid-J v0.20 and the latest pull from the GIT
repository. Both tests were stopped after the host system (in this case
Macbook Pro, Intel Core i7, 64bit, 4 core, 8 GB ram) reported (via top)
that the JVM had consumed resident private address space exceeding 900 MB
with an mx of 1024.
The attachments show the number of messages sent and recieved per second,
followed by the latency of the messages. Where I'm concerned is in the
throughput - QPid 0.20 demonstrates an AMQP 1.0 throughput n the order of
12K messages per second but with a large leak into ram. Your recent work
into optimizing the 1.0 Java code has significantly reduced this, allowing
the tests to run for longer, however the performance is greatly reduced to
about 2000 messages per second.
Looking at VisualVM on OSX (I'm new to VisualVM), in the profiler for the
QPid proces, shows that the largest class allocation in the QPid proces, in
both cases, is of type 'byte[]'. Not having dug into the code yet I'm not
sure what necessarily this means or even if its of any help. However I'm
attaching my test results in addition. I'll also say that while I've
gotten similar results using both client code bases, the tests attached to
this email were collected using the SwiftMQ Java client library.
Sincerely,
Jason
On Wed, Mar 27, 2013 at 8:45 PM, Rob Godfrey <ro...@gmail.com>wrote:
> Hi Jason,
>
> OK... so that's embarrassing :-) The reason that the broker stopped sending
> out messages in my test case was due to a deadlock in the broker side AMQP
> 1.0 codepath [1]. I've applied a fix to the trunk version of the broker,
> which I'll presently be asking for inclusion into the 0.22 release.
>
> Can you see if this also fixes the issue you have been seeing?
>
> Cheers,
> Rob
>
> [1] http://svn.apache.org/viewvc?view=revision&revision=r1461844
>
> On 27 March 2013 20:41, Rob Godfrey <ro...@gmail.com> wrote:
>
> > Hi Jason,
> >
> > just to let you know I am investigating the issue where the broker
> > suddenly stops receiving messages... I can see this too but I haven't yet
> > got to the bottom of why. Apologies for any inconvenience.
> >
> > Cheers,
> > Rob
> >
> >
> > On 27 March 2013 17:51, Jason Barto <ja...@gmail.com> wrote:
> >
> >> Rob,
> >> first let me say thank you so much for your insight and quick responses.
> >> I'm sorry to say that throttling the producers has not completely solved
> >> the problem. I'm back in the office today and have been doing
> additional
> >> testing and I think I may have detected a memory leak in the code that
> >> handles AMQP 1.0 communications.
> >>
> >> My experience thus far:
> >> I downloaded and installed the latest QPid revision using the git
> >> directions on the website. With the Java QPid up and running I ran my
> >> first test client (AMQP 0-9-1). I sent 60k messages/sec, each with a
> 1KB
> >> payload, for 180 seconds. Using top I monitored the JVMs memory usage
> >> which held steady at 233MB. The code for this client can be found on
> >> pastebin at http://pastebin.com/fwDmhxG9. It uses the rabbitmq-client
> >> JAR
> >> for AMQP 0-9-1.
> >>
> >> Again using 'top' and the latest Java QPid I ran my second test client
> at
> >> 8k messages/sec, each with 1 KB payload, for 180 seconds. The memory
> >> utilization of the JVM reported by 'top' rose steadily until the test
> >> ended, at which time the JVM was using 1.1GB of memory. The code for
> this
> >> second test client is at http://pastebin.com/DNTLCj0y and uses v0.23 of
> >> the
> >> qpid-amqp-1-0-client JAR. It may also be worth noting that at about
> >> second
> >> 147 the consumer just stopped receiving messages, no exceptions were
> >> thrown
> >> on either the client or broker, it simply stopped receiving messages.
> >>
> >> Both tests were throttled to ensure that the consumer would consume in a
> >> timely fashion roughly all messages sent, to avoid any buildup of
> messages
> >> in the broker's queue.
> >>
> >> I will admit that I have far more experience with the RabbitMQ client
> >> library than the QPid library so my hope is that there is something
> >> someone
> >> can spot in my test code that is not being done correctly that will
> >> explain
> >> why the JVM is consuming so much memory. Does acknowledging a message
> >> with
> >> Receiver.acknowledge not sufficiently inform the broker that the message
> >> has been received and can be forgotten?
> >>
> >> On longer test runs, 500+ seconds, the broker eventually consumed all of
> >> its allocated ram and as you mentioned the GC began taking over trying
> to
> >> keep things running.
> >>
> >> Sincerely,
> >> Jason
> >>
> >>
> >> On Tue, Mar 26, 2013 at 12:43 PM, Rob Godfrey <rob.j.godfrey@gmail.com
> >> >wrote:
> >>
> >> > Hi Jason,
> >> >
> >> > On 26 March 2013 13:26, Jason Barto <ja...@gmail.com> wrote:
> >> >
> >> > > Rob,
> >> > > thanks for your quick response - I've consolidated the code into a
> >> single
> >> > > java file and would gladly publish it - frankly being new to AMQP
> 1.0
> >> and
> >> > > its client libraries I'd value the feedback. I think I may have
> >> > determined
> >> > > the reason for the slow down - not sure why it didn't occur to me
> >> earlier
> >> > > to be honest.
> >> > >
> >> > > I have 2 threads, a Sender and Receiver. Both are trying to
> produce /
> >> > > consume as rapidly as possible. Testing on my laptop the Sender is
> >> > > publishing on average 20k messages / sec while the receiver seems to
> >> top
> >> > > out at 15K msg/sec, leading to an inevitable backlog of messages
> >> building
> >> > > up on the broker. When the queue on the broker is holding around
> 500k
> >> > > messages (after about 90 seconds of testing) the performance of the
> >> > broker
> >> > > drops dramatically until both producer / consumer can only send /
> >> receive
> >> > > about 200 msg/s.
> >> > >
> >> > >
> >> >
> >> > OK - great (sort of) - we can understand the problem - basically the
> >> broker
> >> > is not sufficiently pushing back on the producing client when the
> queue
> >> > becomes over full.
> >> >
> >> > The broker can be used to enforce flow control when using earlier
> >> versions
> >> > of the protocol (0-8, 0-9, 0-9-1, 0-10) but it's entirely possible
> that
> >> > this is not yet implemented in the 1-0 codepath (I shall have to go
> >> > check... if it is not I'll attempt to implement it shortly).
> >> >
> >> > In general the broker performance shouldn't degrade until it starts
> >> running
> >> > out of memory, whereupon the garbage collector will start dominating
> the
> >> > performance.
> >> >
> >> >
> >> > > If I put a limit on the number of messages / sec sent by the
> producer
> >> the
> >> > > problem goes away, ie the receiver can consume the messages at a
> rate
> >> to
> >> > > prevent a backlog of messages. I'd be happy to create a JIRA
> request
> >> if
> >> > > you feel one is necessary; is there perhaps a datastore I could
> >> configure
> >> > > QPid to use that would help the system to better deal with the
> growing
> >> > > backlog mentioned? I ask as a mitigation technique. I'll be
> >> becoming a
> >> > > sysadmin for an enterprise-level message broker that will be
> expected
> >> to
> >> > > handle 1.75B messages per day (roughly 20k/s). If for some reason
> the
> >> > > messages are not being consumed at the same rate they're being
> >> produced -
> >> > > what options do I have to ensure that QPid's performance is
> >> unaffected?
> >> > >
> >> > >
> >> > I guess the question is how large do you want to let the queue grow
> >> before
> >> > you have to enforce some sort of restriction on the sender? Adding
> >> > flow-to-disk style capabilities often just makes the problem worse
> >> because
> >> > the consumer starts to become limited by the speed at which messages
> >> can be
> >> > read from disk, which may be lower than it's optimal rate... so you
> >> start
> >> > seeing even more queue growth. I've seen people who want to be able
> to
> >> > handle spikes in queue growth of up to several million messages... and
> >> the
> >> > answer to their problem was just to stuff a lot more RAM into their
> >> machine
> >> > :)
> >> >
> >> > As an aside are you expecting these 20K/s messages to be AMQP 1.0?
> As I
> >> > mentioned in my last mail, I've not yet done *any* perf tuning on the
> >> > AMQP1.0 code (JMS client or broker)... Though your question yesterday
> >> has
> >> > forced me to dig up my old copy of JProfiler and run it on the broker
> >> for
> >> > the first time today :-)
> >> >
> >> > Cheers,
> >> > Rob
> >> >
> >> >
> >> > > Sincerely,
> >> > > Jason
> >> > >
> >> > >
> >> > > On Mon, Mar 25, 2013 at 7:08 PM, Rob Godfrey <
> rob.j.godfrey@gmail.com
> >> > > >wrote:
> >> > >
> >> > > > Hi Jason,
> >> > > >
> >> > > > the AMQP 1-0 support in the java broker is somewhat experimental
> at
> >> > > > this stage (and no perf tuning has been carried out), however it
> >> > > > definitely seems like you've found a bug. Can you share some more
> >> > > > details of your test (perhaps raise a JIRA and attach the test
> >> code?)
> >> > > > and I'll try to look into it ASAP - hopefully that way we can get
> a
> >> > > > fix in for 0.22,
> >> > > >
> >> > > > Thanks,
> >> > > > Rob
> >> > > >
> >> > > > On 25 March 2013 19:20, Jason Barto <ja...@gmail.com>
> >> wrote:
> >> > > > > Using amqp 0.9.1 I'm getting an average performance of 70k
> >> messages
> >> > per
> >> > > > > second throughput during an hour long test.
> >> > > > >
> >> > > > > When using amqp 1.0 I receive similar but smaller throughput
> >> numbers
> >> > > > > however after about 90 seconds of testing this quickly begins to
> >> drop
> >> > > > until
> >> > > > > it gets to 10s or 100s of messages per second and a significant
> >> > backlog
> >> > > > of
> >> > > > > messages begins building.
> >> > > > >
> >> > > > > Can anyone explain this drastic difference in performance? I am
> >> using
> >> > > > v0.20
> >> > > > > of qpid-j for the brokering.
> >> > > > >
> >> > > > > Sincerely,
> >> > > > > Jason
> >> > > >
> >> > > >
> >> ---------------------------------------------------------------------
> >> > > > To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> >> > > > For additional commands, e-mail: users-help@qpid.apache.org
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> >
> >
>
Re: Performance degradation over time during amqp 1.0 stress test
Posted by Rob Godfrey <ro...@gmail.com>.
Hi Jason,
OK... so that's embarrassing :-) The reason that the broker stopped sending
out messages in my test case was due to a deadlock in the broker side AMQP
1.0 codepath [1]. I've applied a fix to the trunk version of the broker,
which I'll presently be asking for inclusion into the 0.22 release.
Can you see if this also fixes the issue you have been seeing?
Cheers,
Rob
[1] http://svn.apache.org/viewvc?view=revision&revision=r1461844
On 27 March 2013 20:41, Rob Godfrey <ro...@gmail.com> wrote:
> Hi Jason,
>
> just to let you know I am investigating the issue where the broker
> suddenly stops receiving messages... I can see this too but I haven't yet
> got to the bottom of why. Apologies for any inconvenience.
>
> Cheers,
> Rob
>
>
> On 27 March 2013 17:51, Jason Barto <ja...@gmail.com> wrote:
>
>> Rob,
>> first let me say thank you so much for your insight and quick responses.
>> I'm sorry to say that throttling the producers has not completely solved
>> the problem. I'm back in the office today and have been doing additional
>> testing and I think I may have detected a memory leak in the code that
>> handles AMQP 1.0 communications.
>>
>> My experience thus far:
>> I downloaded and installed the latest QPid revision using the git
>> directions on the website. With the Java QPid up and running I ran my
>> first test client (AMQP 0-9-1). I sent 60k messages/sec, each with a 1KB
>> payload, for 180 seconds. Using top I monitored the JVMs memory usage
>> which held steady at 233MB. The code for this client can be found on
>> pastebin at http://pastebin.com/fwDmhxG9. It uses the rabbitmq-client
>> JAR
>> for AMQP 0-9-1.
>>
>> Again using 'top' and the latest Java QPid I ran my second test client at
>> 8k messages/sec, each with 1 KB payload, for 180 seconds. The memory
>> utilization of the JVM reported by 'top' rose steadily until the test
>> ended, at which time the JVM was using 1.1GB of memory. The code for this
>> second test client is at http://pastebin.com/DNTLCj0y and uses v0.23 of
>> the
>> qpid-amqp-1-0-client JAR. It may also be worth noting that at about
>> second
>> 147 the consumer just stopped receiving messages, no exceptions were
>> thrown
>> on either the client or broker, it simply stopped receiving messages.
>>
>> Both tests were throttled to ensure that the consumer would consume in a
>> timely fashion roughly all messages sent, to avoid any buildup of messages
>> in the broker's queue.
>>
>> I will admit that I have far more experience with the RabbitMQ client
>> library than the QPid library so my hope is that there is something
>> someone
>> can spot in my test code that is not being done correctly that will
>> explain
>> why the JVM is consuming so much memory. Does acknowledging a message
>> with
>> Receiver.acknowledge not sufficiently inform the broker that the message
>> has been received and can be forgotten?
>>
>> On longer test runs, 500+ seconds, the broker eventually consumed all of
>> its allocated ram and as you mentioned the GC began taking over trying to
>> keep things running.
>>
>> Sincerely,
>> Jason
>>
>>
>> On Tue, Mar 26, 2013 at 12:43 PM, Rob Godfrey <rob.j.godfrey@gmail.com
>> >wrote:
>>
>> > Hi Jason,
>> >
>> > On 26 March 2013 13:26, Jason Barto <ja...@gmail.com> wrote:
>> >
>> > > Rob,
>> > > thanks for your quick response - I've consolidated the code into a
>> single
>> > > java file and would gladly publish it - frankly being new to AMQP 1.0
>> and
>> > > its client libraries I'd value the feedback. I think I may have
>> > determined
>> > > the reason for the slow down - not sure why it didn't occur to me
>> earlier
>> > > to be honest.
>> > >
>> > > I have 2 threads, a Sender and Receiver. Both are trying to produce /
>> > > consume as rapidly as possible. Testing on my laptop the Sender is
>> > > publishing on average 20k messages / sec while the receiver seems to
>> top
>> > > out at 15K msg/sec, leading to an inevitable backlog of messages
>> building
>> > > up on the broker. When the queue on the broker is holding around 500k
>> > > messages (after about 90 seconds of testing) the performance of the
>> > broker
>> > > drops dramatically until both producer / consumer can only send /
>> receive
>> > > about 200 msg/s.
>> > >
>> > >
>> >
>> > OK - great (sort of) - we can understand the problem - basically the
>> broker
>> > is not sufficiently pushing back on the producing client when the queue
>> > becomes over full.
>> >
>> > The broker can be used to enforce flow control when using earlier
>> versions
>> > of the protocol (0-8, 0-9, 0-9-1, 0-10) but it's entirely possible that
>> > this is not yet implemented in the 1-0 codepath (I shall have to go
>> > check... if it is not I'll attempt to implement it shortly).
>> >
>> > In general the broker performance shouldn't degrade until it starts
>> running
>> > out of memory, whereupon the garbage collector will start dominating the
>> > performance.
>> >
>> >
>> > > If I put a limit on the number of messages / sec sent by the producer
>> the
>> > > problem goes away, ie the receiver can consume the messages at a rate
>> to
>> > > prevent a backlog of messages. I'd be happy to create a JIRA request
>> if
>> > > you feel one is necessary; is there perhaps a datastore I could
>> configure
>> > > QPid to use that would help the system to better deal with the growing
>> > > backlog mentioned? I ask as a mitigation technique. I'll be
>> becoming a
>> > > sysadmin for an enterprise-level message broker that will be expected
>> to
>> > > handle 1.75B messages per day (roughly 20k/s). If for some reason the
>> > > messages are not being consumed at the same rate they're being
>> produced -
>> > > what options do I have to ensure that QPid's performance is
>> unaffected?
>> > >
>> > >
>> > I guess the question is how large do you want to let the queue grow
>> before
>> > you have to enforce some sort of restriction on the sender? Adding
>> > flow-to-disk style capabilities often just makes the problem worse
>> because
>> > the consumer starts to become limited by the speed at which messages
>> can be
>> > read from disk, which may be lower than it's optimal rate... so you
>> start
>> > seeing even more queue growth. I've seen people who want to be able to
>> > handle spikes in queue growth of up to several million messages... and
>> the
>> > answer to their problem was just to stuff a lot more RAM into their
>> machine
>> > :)
>> >
>> > As an aside are you expecting these 20K/s messages to be AMQP 1.0? As I
>> > mentioned in my last mail, I've not yet done *any* perf tuning on the
>> > AMQP1.0 code (JMS client or broker)... Though your question yesterday
>> has
>> > forced me to dig up my old copy of JProfiler and run it on the broker
>> for
>> > the first time today :-)
>> >
>> > Cheers,
>> > Rob
>> >
>> >
>> > > Sincerely,
>> > > Jason
>> > >
>> > >
>> > > On Mon, Mar 25, 2013 at 7:08 PM, Rob Godfrey <rob.j.godfrey@gmail.com
>> > > >wrote:
>> > >
>> > > > Hi Jason,
>> > > >
>> > > > the AMQP 1-0 support in the java broker is somewhat experimental at
>> > > > this stage (and no perf tuning has been carried out), however it
>> > > > definitely seems like you've found a bug. Can you share some more
>> > > > details of your test (perhaps raise a JIRA and attach the test
>> code?)
>> > > > and I'll try to look into it ASAP - hopefully that way we can get a
>> > > > fix in for 0.22,
>> > > >
>> > > > Thanks,
>> > > > Rob
>> > > >
>> > > > On 25 March 2013 19:20, Jason Barto <ja...@gmail.com>
>> wrote:
>> > > > > Using amqp 0.9.1 I'm getting an average performance of 70k
>> messages
>> > per
>> > > > > second throughput during an hour long test.
>> > > > >
>> > > > > When using amqp 1.0 I receive similar but smaller throughput
>> numbers
>> > > > > however after about 90 seconds of testing this quickly begins to
>> drop
>> > > > until
>> > > > > it gets to 10s or 100s of messages per second and a significant
>> > backlog
>> > > > of
>> > > > > messages begins building.
>> > > > >
>> > > > > Can anyone explain this drastic difference in performance? I am
>> using
>> > > > v0.20
>> > > > > of qpid-j for the brokering.
>> > > > >
>> > > > > Sincerely,
>> > > > > Jason
>> > > >
>> > > >
>> ---------------------------------------------------------------------
>> > > > To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
>> > > > For additional commands, e-mail: users-help@qpid.apache.org
>> > > >
>> > > >
>> > >
>> >
>>
>
>
Re: Performance degradation over time during amqp 1.0 stress test
Posted by Rob Godfrey <ro...@gmail.com>.
Hi Jason,
just to let you know I am investigating the issue where the broker suddenly
stops receiving messages... I can see this too but I haven't yet got to the
bottom of why. Apologies for any inconvenience.
Cheers,
Rob
On 27 March 2013 17:51, Jason Barto <ja...@gmail.com> wrote:
> Rob,
> first let me say thank you so much for your insight and quick responses.
> I'm sorry to say that throttling the producers has not completely solved
> the problem. I'm back in the office today and have been doing additional
> testing and I think I may have detected a memory leak in the code that
> handles AMQP 1.0 communications.
>
> My experience thus far:
> I downloaded and installed the latest QPid revision using the git
> directions on the website. With the Java QPid up and running I ran my
> first test client (AMQP 0-9-1). I sent 60k messages/sec, each with a 1KB
> payload, for 180 seconds. Using top I monitored the JVMs memory usage
> which held steady at 233MB. The code for this client can be found on
> pastebin at http://pastebin.com/fwDmhxG9. It uses the rabbitmq-client JAR
> for AMQP 0-9-1.
>
> Again using 'top' and the latest Java QPid I ran my second test client at
> 8k messages/sec, each with 1 KB payload, for 180 seconds. The memory
> utilization of the JVM reported by 'top' rose steadily until the test
> ended, at which time the JVM was using 1.1GB of memory. The code for this
> second test client is at http://pastebin.com/DNTLCj0y and uses v0.23 of
> the
> qpid-amqp-1-0-client JAR. It may also be worth noting that at about second
> 147 the consumer just stopped receiving messages, no exceptions were thrown
> on either the client or broker, it simply stopped receiving messages.
>
> Both tests were throttled to ensure that the consumer would consume in a
> timely fashion roughly all messages sent, to avoid any buildup of messages
> in the broker's queue.
>
> I will admit that I have far more experience with the RabbitMQ client
> library than the QPid library so my hope is that there is something someone
> can spot in my test code that is not being done correctly that will explain
> why the JVM is consuming so much memory. Does acknowledging a message with
> Receiver.acknowledge not sufficiently inform the broker that the message
> has been received and can be forgotten?
>
> On longer test runs, 500+ seconds, the broker eventually consumed all of
> its allocated ram and as you mentioned the GC began taking over trying to
> keep things running.
>
> Sincerely,
> Jason
>
>
> On Tue, Mar 26, 2013 at 12:43 PM, Rob Godfrey <rob.j.godfrey@gmail.com
> >wrote:
>
> > Hi Jason,
> >
> > On 26 March 2013 13:26, Jason Barto <ja...@gmail.com> wrote:
> >
> > > Rob,
> > > thanks for your quick response - I've consolidated the code into a
> single
> > > java file and would gladly publish it - frankly being new to AMQP 1.0
> and
> > > its client libraries I'd value the feedback. I think I may have
> > determined
> > > the reason for the slow down - not sure why it didn't occur to me
> earlier
> > > to be honest.
> > >
> > > I have 2 threads, a Sender and Receiver. Both are trying to produce /
> > > consume as rapidly as possible. Testing on my laptop the Sender is
> > > publishing on average 20k messages / sec while the receiver seems to
> top
> > > out at 15K msg/sec, leading to an inevitable backlog of messages
> building
> > > up on the broker. When the queue on the broker is holding around 500k
> > > messages (after about 90 seconds of testing) the performance of the
> > broker
> > > drops dramatically until both producer / consumer can only send /
> receive
> > > about 200 msg/s.
> > >
> > >
> >
> > OK - great (sort of) - we can understand the problem - basically the
> broker
> > is not sufficiently pushing back on the producing client when the queue
> > becomes over full.
> >
> > The broker can be used to enforce flow control when using earlier
> versions
> > of the protocol (0-8, 0-9, 0-9-1, 0-10) but it's entirely possible that
> > this is not yet implemented in the 1-0 codepath (I shall have to go
> > check... if it is not I'll attempt to implement it shortly).
> >
> > In general the broker performance shouldn't degrade until it starts
> running
> > out of memory, whereupon the garbage collector will start dominating the
> > performance.
> >
> >
> > > If I put a limit on the number of messages / sec sent by the producer
> the
> > > problem goes away, ie the receiver can consume the messages at a rate
> to
> > > prevent a backlog of messages. I'd be happy to create a JIRA request
> if
> > > you feel one is necessary; is there perhaps a datastore I could
> configure
> > > QPid to use that would help the system to better deal with the growing
> > > backlog mentioned? I ask as a mitigation technique. I'll be becoming
> a
> > > sysadmin for an enterprise-level message broker that will be expected
> to
> > > handle 1.75B messages per day (roughly 20k/s). If for some reason the
> > > messages are not being consumed at the same rate they're being
> produced -
> > > what options do I have to ensure that QPid's performance is unaffected?
> > >
> > >
> > I guess the question is how large do you want to let the queue grow
> before
> > you have to enforce some sort of restriction on the sender? Adding
> > flow-to-disk style capabilities often just makes the problem worse
> because
> > the consumer starts to become limited by the speed at which messages can
> be
> > read from disk, which may be lower than it's optimal rate... so you start
> > seeing even more queue growth. I've seen people who want to be able to
> > handle spikes in queue growth of up to several million messages... and
> the
> > answer to their problem was just to stuff a lot more RAM into their
> machine
> > :)
> >
> > As an aside are you expecting these 20K/s messages to be AMQP 1.0? As I
> > mentioned in my last mail, I've not yet done *any* perf tuning on the
> > AMQP1.0 code (JMS client or broker)... Though your question yesterday has
> > forced me to dig up my old copy of JProfiler and run it on the broker for
> > the first time today :-)
> >
> > Cheers,
> > Rob
> >
> >
> > > Sincerely,
> > > Jason
> > >
> > >
> > > On Mon, Mar 25, 2013 at 7:08 PM, Rob Godfrey <rob.j.godfrey@gmail.com
> > > >wrote:
> > >
> > > > Hi Jason,
> > > >
> > > > the AMQP 1-0 support in the java broker is somewhat experimental at
> > > > this stage (and no perf tuning has been carried out), however it
> > > > definitely seems like you've found a bug. Can you share some more
> > > > details of your test (perhaps raise a JIRA and attach the test code?)
> > > > and I'll try to look into it ASAP - hopefully that way we can get a
> > > > fix in for 0.22,
> > > >
> > > > Thanks,
> > > > Rob
> > > >
> > > > On 25 March 2013 19:20, Jason Barto <ja...@gmail.com> wrote:
> > > > > Using amqp 0.9.1 I'm getting an average performance of 70k messages
> > per
> > > > > second throughput during an hour long test.
> > > > >
> > > > > When using amqp 1.0 I receive similar but smaller throughput
> numbers
> > > > > however after about 90 seconds of testing this quickly begins to
> drop
> > > > until
> > > > > it gets to 10s or 100s of messages per second and a significant
> > backlog
> > > > of
> > > > > messages begins building.
> > > > >
> > > > > Can anyone explain this drastic difference in performance? I am
> using
> > > > v0.20
> > > > > of qpid-j for the brokering.
> > > > >
> > > > > Sincerely,
> > > > > Jason
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> > > > For additional commands, e-mail: users-help@qpid.apache.org
> > > >
> > > >
> > >
> >
>
Re: Performance degradation over time during amqp 1.0 stress test
Posted by Jason Barto <ja...@gmail.com>.
Rob,
first let me say thank you so much for your insight and quick responses.
I'm sorry to say that throttling the producers has not completely solved
the problem. I'm back in the office today and have been doing additional
testing and I think I may have detected a memory leak in the code that
handles AMQP 1.0 communications.
My experience thus far:
I downloaded and installed the latest QPid revision using the git
directions on the website. With the Java QPid up and running I ran my
first test client (AMQP 0-9-1). I sent 60k messages/sec, each with a 1KB
payload, for 180 seconds. Using top I monitored the JVMs memory usage
which held steady at 233MB. The code for this client can be found on
pastebin at http://pastebin.com/fwDmhxG9. It uses the rabbitmq-client JAR
for AMQP 0-9-1.
Again using 'top' and the latest Java QPid I ran my second test client at
8k messages/sec, each with 1 KB payload, for 180 seconds. The memory
utilization of the JVM reported by 'top' rose steadily until the test
ended, at which time the JVM was using 1.1GB of memory. The code for this
second test client is at http://pastebin.com/DNTLCj0y and uses v0.23 of the
qpid-amqp-1-0-client JAR. It may also be worth noting that at about second
147 the consumer just stopped receiving messages, no exceptions were thrown
on either the client or broker, it simply stopped receiving messages.
Both tests were throttled to ensure that the consumer would consume in a
timely fashion roughly all messages sent, to avoid any buildup of messages
in the broker's queue.
I will admit that I have far more experience with the RabbitMQ client
library than the QPid library so my hope is that there is something someone
can spot in my test code that is not being done correctly that will explain
why the JVM is consuming so much memory. Does acknowledging a message with
Receiver.acknowledge not sufficiently inform the broker that the message
has been received and can be forgotten?
On longer test runs, 500+ seconds, the broker eventually consumed all of
its allocated ram and as you mentioned the GC began taking over trying to
keep things running.
Sincerely,
Jason
On Tue, Mar 26, 2013 at 12:43 PM, Rob Godfrey <ro...@gmail.com>wrote:
> Hi Jason,
>
> On 26 March 2013 13:26, Jason Barto <ja...@gmail.com> wrote:
>
> > Rob,
> > thanks for your quick response - I've consolidated the code into a single
> > java file and would gladly publish it - frankly being new to AMQP 1.0 and
> > its client libraries I'd value the feedback. I think I may have
> determined
> > the reason for the slow down - not sure why it didn't occur to me earlier
> > to be honest.
> >
> > I have 2 threads, a Sender and Receiver. Both are trying to produce /
> > consume as rapidly as possible. Testing on my laptop the Sender is
> > publishing on average 20k messages / sec while the receiver seems to top
> > out at 15K msg/sec, leading to an inevitable backlog of messages building
> > up on the broker. When the queue on the broker is holding around 500k
> > messages (after about 90 seconds of testing) the performance of the
> broker
> > drops dramatically until both producer / consumer can only send / receive
> > about 200 msg/s.
> >
> >
>
> OK - great (sort of) - we can understand the problem - basically the broker
> is not sufficiently pushing back on the producing client when the queue
> becomes over full.
>
> The broker can be used to enforce flow control when using earlier versions
> of the protocol (0-8, 0-9, 0-9-1, 0-10) but it's entirely possible that
> this is not yet implemented in the 1-0 codepath (I shall have to go
> check... if it is not I'll attempt to implement it shortly).
>
> In general the broker performance shouldn't degrade until it starts running
> out of memory, whereupon the garbage collector will start dominating the
> performance.
>
>
> > If I put a limit on the number of messages / sec sent by the producer the
> > problem goes away, ie the receiver can consume the messages at a rate to
> > prevent a backlog of messages. I'd be happy to create a JIRA request if
> > you feel one is necessary; is there perhaps a datastore I could configure
> > QPid to use that would help the system to better deal with the growing
> > backlog mentioned? I ask as a mitigation technique. I'll be becoming a
> > sysadmin for an enterprise-level message broker that will be expected to
> > handle 1.75B messages per day (roughly 20k/s). If for some reason the
> > messages are not being consumed at the same rate they're being produced -
> > what options do I have to ensure that QPid's performance is unaffected?
> >
> >
> I guess the question is how large do you want to let the queue grow before
> you have to enforce some sort of restriction on the sender? Adding
> flow-to-disk style capabilities often just makes the problem worse because
> the consumer starts to become limited by the speed at which messages can be
> read from disk, which may be lower than it's optimal rate... so you start
> seeing even more queue growth. I've seen people who want to be able to
> handle spikes in queue growth of up to several million messages... and the
> answer to their problem was just to stuff a lot more RAM into their machine
> :)
>
> As an aside are you expecting these 20K/s messages to be AMQP 1.0? As I
> mentioned in my last mail, I've not yet done *any* perf tuning on the
> AMQP1.0 code (JMS client or broker)... Though your question yesterday has
> forced me to dig up my old copy of JProfiler and run it on the broker for
> the first time today :-)
>
> Cheers,
> Rob
>
>
> > Sincerely,
> > Jason
> >
> >
> > On Mon, Mar 25, 2013 at 7:08 PM, Rob Godfrey <rob.j.godfrey@gmail.com
> > >wrote:
> >
> > > Hi Jason,
> > >
> > > the AMQP 1-0 support in the java broker is somewhat experimental at
> > > this stage (and no perf tuning has been carried out), however it
> > > definitely seems like you've found a bug. Can you share some more
> > > details of your test (perhaps raise a JIRA and attach the test code?)
> > > and I'll try to look into it ASAP - hopefully that way we can get a
> > > fix in for 0.22,
> > >
> > > Thanks,
> > > Rob
> > >
> > > On 25 March 2013 19:20, Jason Barto <ja...@gmail.com> wrote:
> > > > Using amqp 0.9.1 I'm getting an average performance of 70k messages
> per
> > > > second throughput during an hour long test.
> > > >
> > > > When using amqp 1.0 I receive similar but smaller throughput numbers
> > > > however after about 90 seconds of testing this quickly begins to drop
> > > until
> > > > it gets to 10s or 100s of messages per second and a significant
> backlog
> > > of
> > > > messages begins building.
> > > >
> > > > Can anyone explain this drastic difference in performance? I am using
> > > v0.20
> > > > of qpid-j for the brokering.
> > > >
> > > > Sincerely,
> > > > Jason
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> > > For additional commands, e-mail: users-help@qpid.apache.org
> > >
> > >
> >
>
Re: Performance degradation over time during amqp 1.0 stress test
Posted by Rob Godfrey <ro...@gmail.com>.
Hi Jason,
On 26 March 2013 13:26, Jason Barto <ja...@gmail.com> wrote:
> Rob,
> thanks for your quick response - I've consolidated the code into a single
> java file and would gladly publish it - frankly being new to AMQP 1.0 and
> its client libraries I'd value the feedback. I think I may have determined
> the reason for the slow down - not sure why it didn't occur to me earlier
> to be honest.
>
> I have 2 threads, a Sender and Receiver. Both are trying to produce /
> consume as rapidly as possible. Testing on my laptop the Sender is
> publishing on average 20k messages / sec while the receiver seems to top
> out at 15K msg/sec, leading to an inevitable backlog of messages building
> up on the broker. When the queue on the broker is holding around 500k
> messages (after about 90 seconds of testing) the performance of the broker
> drops dramatically until both producer / consumer can only send / receive
> about 200 msg/s.
>
>
OK - great (sort of) - we can understand the problem - basically the broker
is not sufficiently pushing back on the producing client when the queue
becomes over full.
The broker can be used to enforce flow control when using earlier versions
of the protocol (0-8, 0-9, 0-9-1, 0-10) but it's entirely possible that
this is not yet implemented in the 1-0 codepath (I shall have to go
check... if it is not I'll attempt to implement it shortly).
In general the broker performance shouldn't degrade until it starts running
out of memory, whereupon the garbage collector will start dominating the
performance.
> If I put a limit on the number of messages / sec sent by the producer the
> problem goes away, ie the receiver can consume the messages at a rate to
> prevent a backlog of messages. I'd be happy to create a JIRA request if
> you feel one is necessary; is there perhaps a datastore I could configure
> QPid to use that would help the system to better deal with the growing
> backlog mentioned? I ask as a mitigation technique. I'll be becoming a
> sysadmin for an enterprise-level message broker that will be expected to
> handle 1.75B messages per day (roughly 20k/s). If for some reason the
> messages are not being consumed at the same rate they're being produced -
> what options do I have to ensure that QPid's performance is unaffected?
>
>
I guess the question is how large do you want to let the queue grow before
you have to enforce some sort of restriction on the sender? Adding
flow-to-disk style capabilities often just makes the problem worse because
the consumer starts to become limited by the speed at which messages can be
read from disk, which may be lower than it's optimal rate... so you start
seeing even more queue growth. I've seen people who want to be able to
handle spikes in queue growth of up to several million messages... and the
answer to their problem was just to stuff a lot more RAM into their machine
:)
As an aside are you expecting these 20K/s messages to be AMQP 1.0? As I
mentioned in my last mail, I've not yet done *any* perf tuning on the
AMQP1.0 code (JMS client or broker)... Though your question yesterday has
forced me to dig up my old copy of JProfiler and run it on the broker for
the first time today :-)
Cheers,
Rob
> Sincerely,
> Jason
>
>
> On Mon, Mar 25, 2013 at 7:08 PM, Rob Godfrey <rob.j.godfrey@gmail.com
> >wrote:
>
> > Hi Jason,
> >
> > the AMQP 1-0 support in the java broker is somewhat experimental at
> > this stage (and no perf tuning has been carried out), however it
> > definitely seems like you've found a bug. Can you share some more
> > details of your test (perhaps raise a JIRA and attach the test code?)
> > and I'll try to look into it ASAP - hopefully that way we can get a
> > fix in for 0.22,
> >
> > Thanks,
> > Rob
> >
> > On 25 March 2013 19:20, Jason Barto <ja...@gmail.com> wrote:
> > > Using amqp 0.9.1 I'm getting an average performance of 70k messages per
> > > second throughput during an hour long test.
> > >
> > > When using amqp 1.0 I receive similar but smaller throughput numbers
> > > however after about 90 seconds of testing this quickly begins to drop
> > until
> > > it gets to 10s or 100s of messages per second and a significant backlog
> > of
> > > messages begins building.
> > >
> > > Can anyone explain this drastic difference in performance? I am using
> > v0.20
> > > of qpid-j for the brokering.
> > >
> > > Sincerely,
> > > Jason
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> > For additional commands, e-mail: users-help@qpid.apache.org
> >
> >
>
Re: Performance degradation over time during amqp 1.0 stress test
Posted by Jason Barto <ja...@gmail.com>.
Rob,
thanks for your quick response - I've consolidated the code into a single
java file and would gladly publish it - frankly being new to AMQP 1.0 and
its client libraries I'd value the feedback. I think I may have determined
the reason for the slow down - not sure why it didn't occur to me earlier
to be honest.
I have 2 threads, a Sender and Receiver. Both are trying to produce /
consume as rapidly as possible. Testing on my laptop the Sender is
publishing on average 20k messages / sec while the receiver seems to top
out at 15K msg/sec, leading to an inevitable backlog of messages building
up on the broker. When the queue on the broker is holding around 500k
messages (after about 90 seconds of testing) the performance of the broker
drops dramatically until both producer / consumer can only send / receive
about 200 msg/s.
If I put a limit on the number of messages / sec sent by the producer the
problem goes away, ie the receiver can consume the messages at a rate to
prevent a backlog of messages. I'd be happy to create a JIRA request if
you feel one is necessary; is there perhaps a datastore I could configure
QPid to use that would help the system to better deal with the growing
backlog mentioned? I ask as a mitigation technique. I'll be becoming a
sysadmin for an enterprise-level message broker that will be expected to
handle 1.75B messages per day (roughly 20k/s). If for some reason the
messages are not being consumed at the same rate they're being produced -
what options do I have to ensure that QPid's performance is unaffected?
Sincerely,
Jason
On Mon, Mar 25, 2013 at 7:08 PM, Rob Godfrey <ro...@gmail.com>wrote:
> Hi Jason,
>
> the AMQP 1-0 support in the java broker is somewhat experimental at
> this stage (and no perf tuning has been carried out), however it
> definitely seems like you've found a bug. Can you share some more
> details of your test (perhaps raise a JIRA and attach the test code?)
> and I'll try to look into it ASAP - hopefully that way we can get a
> fix in for 0.22,
>
> Thanks,
> Rob
>
> On 25 March 2013 19:20, Jason Barto <ja...@gmail.com> wrote:
> > Using amqp 0.9.1 I'm getting an average performance of 70k messages per
> > second throughput during an hour long test.
> >
> > When using amqp 1.0 I receive similar but smaller throughput numbers
> > however after about 90 seconds of testing this quickly begins to drop
> until
> > it gets to 10s or 100s of messages per second and a significant backlog
> of
> > messages begins building.
> >
> > Can anyone explain this drastic difference in performance? I am using
> v0.20
> > of qpid-j for the brokering.
> >
> > Sincerely,
> > Jason
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
>
Re: Performance degradation over time during amqp 1.0 stress test
Posted by Rob Godfrey <ro...@gmail.com>.
Hi Jason,
the AMQP 1-0 support in the java broker is somewhat experimental at
this stage (and no perf tuning has been carried out), however it
definitely seems like you've found a bug. Can you share some more
details of your test (perhaps raise a JIRA and attach the test code?)
and I'll try to look into it ASAP - hopefully that way we can get a
fix in for 0.22,
Thanks,
Rob
On 25 March 2013 19:20, Jason Barto <ja...@gmail.com> wrote:
> Using amqp 0.9.1 I'm getting an average performance of 70k messages per
> second throughput during an hour long test.
>
> When using amqp 1.0 I receive similar but smaller throughput numbers
> however after about 90 seconds of testing this quickly begins to drop until
> it gets to 10s or 100s of messages per second and a significant backlog of
> messages begins building.
>
> Can anyone explain this drastic difference in performance? I am using v0.20
> of qpid-j for the brokering.
>
> Sincerely,
> Jason
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org