You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by zaoliu <za...@gmail.com> on 2008/02/21 19:20:44 UTC
Performance issue for ActiveMQ Server
I am tuning the performance of ActiveMQ broker for Queue by using three
separate machines as producer, consumer and server. I got pretty good
throughput for non persistent messages, approximate 20000 msgs/s. For
persistent messages, approximate 4000 megs/s. But when I tested for message
store of the server by using persistent messages, the result is unexpected.
First I use the producer to send the messages to the server and no consumer
yet, so the messages will write to the server message store. The ave rate
for it is 3500/s. And then I stopped the producer and started the consumer
to consume messages. The ave time for consume messages is 50 msgs/s and I
found the rate is keeping going down with time.
Any comments or suggestions?
--
View this message in context: http://www.nabble.com/Performance-issue-for-ActiveMQ-Server-tp15617616s2354p15617616.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Performance issue for ActiveMQ Server
Posted by vc123 <bo...@hotmail.com>.
zaoliu wrote:
>
> I am tuning the performance of ActiveMQ broker for Queue by using three
> separate machines as producer, consumer and server. I got pretty good
> throughput for non persistent messages, approximate 20000 msgs/s. For
> persistent messages, approximate 4000 megs/s. But when I tested for
> message store of the server by using persistent messages, the result is
> unexpected. First I use the producer to send the messages to the server
> and no consumer yet, so the messages will write to the server message
> store. The ave rate for it is 3500/s. And then I stopped the producer and
> started the consumer to consume messages. The ave time for consume
> messages is 50 msgs/s and I found the rate is keeping going down with
> time.
>
> Any comments or suggestions?
>
Depending on your IO subsystem, 50m/s is what one might expect when using
synchronous writes. Any writes other than synchronous make PERSISTENT
message delivery mode pretty meaningless since the writes would go (as they
apparently do in your case) to the filesystem cache and will surely be lost
if the broker crashes. There is no disk on Earth capable of performing 4000
random IOP/s whatever the block size might be. I doubt that you ran your
experiments on a high-end disk array with hundreds of parallel spindles.
So, you need to understand your data flow first and adjust expectation
accordingly.
--
View this message in context: http://www.nabble.com/Performance-issue-for-ActiveMQ-Server-tp15617616s2354p15671312.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Performance issue for ActiveMQ Server
Posted by Ben Chobot <bc...@vigilos.com>.
Rob Davies wrote:
> Writes are usually fast because of the journaling part of the default
> message store
> Which version are you using btw?
I'm using 5.0, and like I said in another email, when writing safely
(syncOnWrite="true") performance isn't all that good without write
caching. How do you define "fast"?
Re: Performance issue for ActiveMQ Server
Posted by Rob Davies <ra...@gmail.com>.
On 21 Feb 2008, at 19:52, Ben Chobot wrote:
> It sounds like you don't have a very firm grasp on what's going on,
> under the covers. If you need persistence, have you tried pulling
> the power cord from your server and verifying you haven't lost
> messages? If you don't care, why do you need persistence? :)
>
> I suspect that what might be happening is that your writes are going
> very fast because they're going into a write buffer somewhere along
> the line. Your reads are not going so fast, because what you're
> trying to read has left the buffer and now has to come off disk, and
> your disk isn't very fast.
>
> zaoliu wrote:
>> I have no idea about the IO systems. The server is using Linux
>> Operating
>> System and has 6 cpus. I find that the speed is getting lower and
>> lower
>> with time. When produce 500000 messages, the speed is 3500/s. But
>> when
>> consumer 700000 messages, the speed get down to avg 1700/s. I
>> think I need
>> to run the test for longer time to get a fair result. When
>> consuming the
>> messages from the server, the speed is getting down much faster. Is
>> 50 msg/s
>> in normal condition?
>>
>>
>> Ben Chobot wrote:
>>
>>> What is the underlying I/O system like for your broker? As I
>>> understand the persistence layer for ActiveMQ, 3,500 durable
>>> messages a second sounds too good to be true, assuming you're
>>> writing them safely.
>>>
>>>
>>>
>
Writes are usually fast because of the journaling part of the default
message store
Which version are you using btw?
cheers,
Rob
Re: Performance issue for ActiveMQ Server
Posted by Zao Liu <za...@cs.indiana.edu>.
I am using Apache ActiveMQ 5.0. To my understand, dispatchSync only helpful
when have slower consumers and using the consumers using the same
connection, different sessions. But in my case, all consumer threads are
using different connections.
Thanks,
Zao
On Fri, Feb 22, 2008 at 1:28 AM, Rob Davies <ra...@gmail.com> wrote:
> You can try setting async dispatch on the consumer - see:
> http://activemq.apache.org/consumer-dispatch-async.html
> A separate task would be used then for dispatching to each consumer
>
> cheers.
>
> Rob
> On 21 Feb 2008, at 22:40, Zao Liu wrote:
>
> > Another issue I met is when I increase the number of consumers for
> > testing
> > non-persistent messages,
> > the throughput also get down very fast. I can't find the reason for
> > it.
> > Each consumer thread is a separate connection to the broker.
> > Below is my result for testing (all using one producer to send
> > messages in a
> > separate JVM):
> > 1 consumer: 12683/s
> > 2 consumers: 11289/s
> > 3 consumers: 9956/s
> > 4 consumers: 8638/s
> > 10 consumers: 5820/s
> >
> > Zao
> >
> >
> > On Thu, Feb 21, 2008 at 2:16 PM, Ben Chobot <bc...@vigilos.com>
> > wrote:
> >
> >> Well, I'm just starting to play with ActiveMQ myself, but just as a
> >> datapoint, I found that when writing every durable message to disk
> >> using
> >> the default Kaha store in 5.0, I maxed out at transferring a
> >> whopping 14
> >> 40KB msgs/s. That's using a single, low-performance 7200RPM drive
> >> without write caching. That's slow, but it's also safe, and it's
> >> probably going to be fast enough for us.
> >>
> >> If anybody has any hints on how to make things go faster while still
> >> being safe, I'd love to hear them.
> >>
> >> Zao Liu wrote:
> >>> Yeah, I do need to test for persistence, but firstly I wan to make
> >>> sure
> >> the
> >>> performance
> >>> is good. The result for consuming durable messages make me
> >>> disappointed.
> >> Is
> >>> any specific
> >>> configuration for consuming durable messages? I do test for
> >>> producing
> >> and
> >>> consuming
> >>> durable messages at the same time, the throughput is also get more
> >>> than
> >>> 3500/s for consuming
> >>> messages. I don't think to consuming messages which has been saved
> >>> to
> >> store
> >>> will be so slow,
> >>> only 50 msgs/s.
> >>>
> >>>
> >>
> >>
>
>
Re: Performance issue for ActiveMQ Server
Posted by Rob Davies <ra...@gmail.com>.
You can try setting async dispatch on the consumer - see: http://activemq.apache.org/consumer-dispatch-async.html
A separate task would be used then for dispatching to each consumer
cheers.
Rob
On 21 Feb 2008, at 22:40, Zao Liu wrote:
> Another issue I met is when I increase the number of consumers for
> testing
> non-persistent messages,
> the throughput also get down very fast. I can't find the reason for
> it.
> Each consumer thread is a separate connection to the broker.
> Below is my result for testing (all using one producer to send
> messages in a
> separate JVM):
> 1 consumer: 12683/s
> 2 consumers: 11289/s
> 3 consumers: 9956/s
> 4 consumers: 8638/s
> 10 consumers: 5820/s
>
> Zao
>
>
> On Thu, Feb 21, 2008 at 2:16 PM, Ben Chobot <bc...@vigilos.com>
> wrote:
>
>> Well, I'm just starting to play with ActiveMQ myself, but just as a
>> datapoint, I found that when writing every durable message to disk
>> using
>> the default Kaha store in 5.0, I maxed out at transferring a
>> whopping 14
>> 40KB msgs/s. That's using a single, low-performance 7200RPM drive
>> without write caching. That's slow, but it's also safe, and it's
>> probably going to be fast enough for us.
>>
>> If anybody has any hints on how to make things go faster while still
>> being safe, I'd love to hear them.
>>
>> Zao Liu wrote:
>>> Yeah, I do need to test for persistence, but firstly I wan to make
>>> sure
>> the
>>> performance
>>> is good. The result for consuming durable messages make me
>>> disappointed.
>> Is
>>> any specific
>>> configuration for consuming durable messages? I do test for
>>> producing
>> and
>>> consuming
>>> durable messages at the same time, the throughput is also get more
>>> than
>>> 3500/s for consuming
>>> messages. I don't think to consuming messages which has been saved
>>> to
>> store
>>> will be so slow,
>>> only 50 msgs/s.
>>>
>>>
>>
>>
Re: Performance issue for ActiveMQ Server
Posted by Zao Liu <za...@cs.indiana.edu>.
Another issue I met is when I increase the number of consumers for testing
non-persistent messages,
the throughput also get down very fast. I can't find the reason for it.
Each consumer thread is a separate connection to the broker.
Below is my result for testing (all using one producer to send messages in a
separate JVM):
1 consumer: 12683/s
2 consumers: 11289/s
3 consumers: 9956/s
4 consumers: 8638/s
10 consumers: 5820/s
Zao
On Thu, Feb 21, 2008 at 2:16 PM, Ben Chobot <bc...@vigilos.com> wrote:
> Well, I'm just starting to play with ActiveMQ myself, but just as a
> datapoint, I found that when writing every durable message to disk using
> the default Kaha store in 5.0, I maxed out at transferring a whopping 14
> 40KB msgs/s. That's using a single, low-performance 7200RPM drive
> without write caching. That's slow, but it's also safe, and it's
> probably going to be fast enough for us.
>
> If anybody has any hints on how to make things go faster while still
> being safe, I'd love to hear them.
>
> Zao Liu wrote:
> > Yeah, I do need to test for persistence, but firstly I wan to make sure
> the
> > performance
> > is good. The result for consuming durable messages make me disappointed.
> Is
> > any specific
> > configuration for consuming durable messages? I do test for producing
> and
> > consuming
> > durable messages at the same time, the throughput is also get more than
> > 3500/s for consuming
> > messages. I don't think to consuming messages which has been saved to
> store
> > will be so slow,
> > only 50 msgs/s.
> >
> >
>
>
Re: Performance issue for ActiveMQ Server
Posted by Ben Chobot <bc...@vigilos.com>.
Well, I'm just starting to play with ActiveMQ myself, but just as a
datapoint, I found that when writing every durable message to disk using
the default Kaha store in 5.0, I maxed out at transferring a whopping 14
40KB msgs/s. That's using a single, low-performance 7200RPM drive
without write caching. That's slow, but it's also safe, and it's
probably going to be fast enough for us.
If anybody has any hints on how to make things go faster while still
being safe, I'd love to hear them.
Zao Liu wrote:
> Yeah, I do need to test for persistence, but firstly I wan to make sure the
> performance
> is good. The result for consuming durable messages make me disappointed. Is
> any specific
> configuration for consuming durable messages? I do test for producing and
> consuming
> durable messages at the same time, the throughput is also get more than
> 3500/s for consuming
> messages. I don't think to consuming messages which has been saved to store
> will be so slow,
> only 50 msgs/s.
>
>
Re: Performance issue for ActiveMQ Server
Posted by Zao Liu <za...@cs.indiana.edu>.
Yeah, I do need to test for persistence, but firstly I wan to make sure the
performance
is good. The result for consuming durable messages make me disappointed. Is
any specific
configuration for consuming durable messages? I do test for producing and
consuming
durable messages at the same time, the throughput is also get more than
3500/s for consuming
messages. I don't think to consuming messages which has been saved to store
will be so slow,
only 50 msgs/s.
Thanks,
Zao Liu
On Thu, Feb 21, 2008 at 11:52 AM, Ben Chobot <bc...@vigilos.com> wrote:
> It sounds like you don't have a very firm grasp on what's going on,
> under the covers. If you need persistence, have you tried pulling the
> power cord from your server and verifying you haven't lost messages? If
> you don't care, why do you need persistence? :)
>
> I suspect that what might be happening is that your writes are going
> very fast because they're going into a write buffer somewhere along the
> line. Your reads are not going so fast, because what you're trying to
> read has left the buffer and now has to come off disk, and your disk
> isn't very fast.
>
> zaoliu wrote:
> > I have no idea about the IO systems. The server is using Linux Operating
> > System and has 6 cpus. I find that the speed is getting lower and lower
> > with time. When produce 500000 messages, the speed is 3500/s. But when
> > consumer 700000 messages, the speed get down to avg 1700/s. I think I
> need
> > to run the test for longer time to get a fair result. When consuming the
> > messages from the server, the speed is getting down much faster. Is 50
> msg/s
> > in normal condition?
> >
> >
> >
> > Ben Chobot wrote:
> >
> >> What is the underlying I/O system like for your broker? As I understand
> >> the persistence layer for ActiveMQ, 3,500 durable messages a second
> >> sounds too good to be true, assuming you're writing them safely.
> >>
> >>
> >>
>
>
Re: Performance issue for ActiveMQ Server
Posted by Ben Chobot <bc...@vigilos.com>.
It sounds like you don't have a very firm grasp on what's going on,
under the covers. If you need persistence, have you tried pulling the
power cord from your server and verifying you haven't lost messages? If
you don't care, why do you need persistence? :)
I suspect that what might be happening is that your writes are going
very fast because they're going into a write buffer somewhere along the
line. Your reads are not going so fast, because what you're trying to
read has left the buffer and now has to come off disk, and your disk
isn't very fast.
zaoliu wrote:
> I have no idea about the IO systems. The server is using Linux Operating
> System and has 6 cpus. I find that the speed is getting lower and lower
> with time. When produce 500000 messages, the speed is 3500/s. But when
> consumer 700000 messages, the speed get down to avg 1700/s. I think I need
> to run the test for longer time to get a fair result. When consuming the
> messages from the server, the speed is getting down much faster. Is 50 msg/s
> in normal condition?
>
>
>
> Ben Chobot wrote:
>
>> What is the underlying I/O system like for your broker? As I understand
>> the persistence layer for ActiveMQ, 3,500 durable messages a second
>> sounds too good to be true, assuming you're writing them safely.
>>
>>
>>
Re: Performance issue for ActiveMQ Server
Posted by zaoliu <za...@gmail.com>.
I have no idea about the IO systems. The server is using Linux Operating
System and has 6 cpus. I find that the speed is getting lower and lower
with time. When produce 500000 messages, the speed is 3500/s. But when
consumer 700000 messages, the speed get down to avg 1700/s. I think I need
to run the test for longer time to get a fair result. When consuming the
messages from the server, the speed is getting down much faster. Is 50 msg/s
in normal condition?
Ben Chobot wrote:
>
> What is the underlying I/O system like for your broker? As I understand
> the persistence layer for ActiveMQ, 3,500 durable messages a second
> sounds too good to be true, assuming you're writing them safely.
>
> zaoliu wrote:
>> I am tuning the performance of ActiveMQ broker for Queue by using three
>> separate machines as producer, consumer and server. I got pretty good
>> throughput for non persistent messages, approximate 20000 msgs/s. For
>> persistent messages, approximate 4000 megs/s. But when I tested for
>> message
>> store of the server by using persistent messages, the result is
>> unexpected.
>> First I use the producer to send the messages to the server and no
>> consumer
>> yet, so the messages will write to the server message store. The ave rate
>> for it is 3500/s. And then I stopped the producer and started the
>> consumer
>> to consume messages. The ave time for consume messages is 50 msgs/s and I
>> found the rate is keeping going down with time.
>>
>> Any comments or suggestions?
>>
>
>
>
--
View this message in context: http://www.nabble.com/Performance-issue-for-ActiveMQ-Server-tp15617616s2354p15618683.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Performance issue for ActiveMQ Server
Posted by Ben Chobot <bc...@vigilos.com>.
What is the underlying I/O system like for your broker? As I understand
the persistence layer for ActiveMQ, 3,500 durable messages a second
sounds too good to be true, assuming you're writing them safely.
zaoliu wrote:
> I am tuning the performance of ActiveMQ broker for Queue by using three
> separate machines as producer, consumer and server. I got pretty good
> throughput for non persistent messages, approximate 20000 msgs/s. For
> persistent messages, approximate 4000 megs/s. But when I tested for message
> store of the server by using persistent messages, the result is unexpected.
> First I use the producer to send the messages to the server and no consumer
> yet, so the messages will write to the server message store. The ave rate
> for it is 3500/s. And then I stopped the producer and started the consumer
> to consume messages. The ave time for consume messages is 50 msgs/s and I
> found the rate is keeping going down with time.
>
> Any comments or suggestions?
>