You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by MassDosage <ma...@gmail.com> on 2009/04/02 19:39:59 UTC
Options for preferring stability over reliability
We are using ActiveMQ quite extensively at Last.fm and are experiencing a
number of issues related to the huge load we are putting on it (at peak
times up to thousands of messages per second). We are using it in a number
of different scenarios but our general use case is that we would much rather
lose hundreds of messages than have deadlocks, out of memory errors and
crashes.
We have turned off producer flow control, set persistence to false, are
using async send, short expiry times on messages etc. Even with this setup
we still regularly (several times per week) run into instances where
ActiveMQ completely fails us - ranging from Out of Memory Exceptions in the
server to blocked Senders to the entire system locking up/freezing with
nothing useful in the log files. These situations force us to restart the
ActiveMQ server regularly and have brought the affected part of our service
to their knees. Ideally this would never happen and in times of heavy load
Senders would not block but instead would drop messages and the server would
do the same instead of running out of memory. Are we missing some
configuration/API options we can use to ensure this behaviour? Does anyone
have any suggestions? I am happy to post our activemq.xml and code samples
if necessary.
I can understand that many of your use cases require reliability and that
you have built much of the system to ensure as few messages get dropped as
possible but we really want the opposite - a system that never goes down or
causes deadlocks and we are willing to lose as many messages as it takes to
allow for this.
--
View this message in context: http://www.nabble.com/Options-for-preferring-stability-over-reliability-tp22852174p22852174.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Options for preferring stability over reliability
Posted by geoffmciver <ge...@team.telstraclear.co.nz>.
MassDosage wrote:
>
> What is this wrapper you are talking about?
>
> I've never seen that "couldn't ping JVM" message so I'm not sure whether
> this is the same issue or not.
>
>
>
ActiveMQ gets "shipped" with Java wrappers http://wrapper.tanukisoftware.org
its a way of daemonising java apps. It forks the JVM process as its child
and keeps tabs on it. If it dies it can restart it.
the underlying issue is the same I think as all wrappers does it restart
activemq. it I run it without wrappers the result is the same as you...
the JVM just hangs with no errors to the logs or console.
--
View this message in context: http://www.nabble.com/Options-for-preferring-stability-over-reliability-tp22852174p23395150.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Options for preferring stability over reliability
Posted by MassDosage <ma...@gmail.com>.
What is this wrapper you are talking about?
I've never seen that "couldn't ping JVM" message so I'm not sure whether
this is the same issue or not.
--
View this message in context: http://www.nabble.com/Options-for-preferring-stability-over-reliability-tp22852174p23384104.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Options for preferring stability over reliability
Posted by geoffmciver <ge...@team.telstraclear.co.nz>.
I too am seeing an issue that activemq freezes completely, Im not sure if it
is under "heavy" load at the time.
I'm using it as a simple queue with one producer and one consumer.
Nothing appears in the logs and wrappers reports that it "couldn't ping JVM
for xx seconds" and restarts ActiveMQ. But sometimes wrappers fails to
restart it and it just sits in a totally hung state.
There are no JVM crash logs or anything in the activemq.log that reports
anything. Its just like it freezes in mid air.
I wouldn't imagine that the load is high as we one have 100's of messages a
second and not 1000's as quoted by many other ActiveMQ users.
With our usage profile it appears to run for 1 week and then crash... It
smealt like a memory leak issue but I would have expected the JVM to report
the fact or have Heap space errors appearing some where.
is there any way to get more verbose logging or would you recommend another
course of action.
I don't want to hijack this thread, I just think that my issue could be the
same.
Thanks
Geoff McIver
MassDosage wrote:
>
> We are using ActiveMQ quite extensively at Last.fm and are experiencing a
> number of issues related to the huge load we are putting on it (at peak
> times up to thousands of messages per second). We are using it in a number
> of different scenarios but our general use case is that we would much
> rather lose hundreds of messages than have deadlocks, out of memory errors
> and crashes.
>
> We have turned off producer flow control, set persistence to false, are
> using async send, short expiry times on messages etc. Even with this setup
> we still regularly (several times per week) run into instances where
> ActiveMQ completely fails us - ranging from Out of Memory Exceptions in
> the server to blocked Senders to the entire system locking up/freezing
> with nothing useful in the log files. These situations force us to restart
> the ActiveMQ server regularly and have brought the affected part of our
> service to their knees. Ideally this would never happen and in times of
> heavy load Senders would not block but instead would drop messages and the
> server would do the same instead of running out of memory. Are we missing
> some configuration/API options we can use to ensure this behaviour? Does
> anyone have any suggestions? I am happy to post our activemq.xml and code
> samples if necessary.
>
> I can understand that many of your use cases require reliability and that
> you have built much of the system to ensure as few messages get dropped as
> possible but we really want the opposite - a system that never goes down
> or causes deadlocks and we are willing to lose as many messages as it
> takes to allow for this.
>
>
--
View this message in context: http://www.nabble.com/Options-for-preferring-stability-over-reliability-tp22852174p23360873.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Options for preferring stability over reliability
Posted by MassDosage <ma...@gmail.com>.
Thanks for the quick response. We are running 2 instances of ActiveMQ - one
is running 5.1.0 and the other is running 5.2.0. The one running 5.1.0 runs
out of memory every now and then and all connected senders go nuts and we
see a *big* increase in CPU on the sender machines (possibly due to high
number of blocked senders?) I will try to a jstack on both the sender
process and the ActiveMQ process next time this happens.
The 5.2.0 setup (see attached activemq.xml) is the one that stops responding
completely with nothing immediately suspicious in the logs. The last time
this happened I did a jstack on our sending application and there were
*many* entries like this:
"pool-1-thread-38" prio=10 tid=0x0000000000aa7800 nid=0x7c6b waiting for
monitor entry [0x0000000044745000..0x0000000044745aa0]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.activemq.ActiveMQSession.send(ActiveMQSession.java:1599)
- waiting to lock <0x00002b0f421e1e10> (a java.lang.Object)
at
org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:227)
at
org.apache.activemq.ActiveMQMessageProducerSupport.send(ActiveMQMessageProducerSupport.java:241)
at fm.last.jms.Publisher.publish(Publisher.java:82)
.....
The sending machine's CPU and memory were maxed out, probably due to the JVM
having to handle all these thousands of blocked threads. We are using
sendFailIfNoSpace but I had no idea that was mutually exclusive with turning
flow control off. We would like to avoid flow control if we can as this has
caused us problems in the past.
If there is anything else I can provide you with please let me know. I will
try get jstack output for the ActiveMQ processes the next time we have
issues. I'm afraid this isn't something that we can't reproduce in a unit
test as it really seems to happen unpredictably when the system has been
running for a while. It's not even that it always happens during peak load
times. http://www.nabble.com/file/p22858692/activemq.xml activemq.xml
--
View this message in context: http://www.nabble.com/Options-for-preferring-stability-over-reliability-tp22852174p22858692.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Options for preferring stability over reliability
Posted by Gary Tully <ga...@gmail.com>.
please post as much information as you can w.r.t number of topics, queues
and xml configuration. If you can produce a junit test case that
demonstrates the behaviour you are experiencing that would be fantastic.
Using systemUsage it should be possible to ensure memory is never exhausted
and messages are spooled to temp storage on disk.
An alternative strategy is to enable sendFailIfNoSpace on systemUsage which
will ensure that there is no sender block when a usage limit is reached.
Flow control needs to be enabled to make use of this feature and you
producers will need to handle the JMSException that may be thrown.
What version are you using?
2009/4/2 MassDosage <ma...@gmail.com>
>
> We are using ActiveMQ quite extensively at Last.fm and are experiencing a
> number of issues related to the huge load we are putting on it (at peak
> times up to thousands of messages per second). We are using it in a number
> of different scenarios but our general use case is that we would much
> rather
> lose hundreds of messages than have deadlocks, out of memory errors and
> crashes.
>
> We have turned off producer flow control, set persistence to false, are
> using async send, short expiry times on messages etc. Even with this setup
> we still regularly (several times per week) run into instances where
> ActiveMQ completely fails us - ranging from Out of Memory Exceptions in the
> server to blocked Senders to the entire system locking up/freezing with
> nothing useful in the log files. These situations force us to restart the
> ActiveMQ server regularly and have brought the affected part of our service
> to their knees. Ideally this would never happen and in times of heavy load
> Senders would not block but instead would drop messages and the server
> would
> do the same instead of running out of memory. Are we missing some
> configuration/API options we can use to ensure this behaviour? Does anyone
> have any suggestions? I am happy to post our activemq.xml and code samples
> if necessary.
>
> I can understand that many of your use cases require reliability and that
> you have built much of the system to ensure as few messages get dropped as
> possible but we really want the opposite - a system that never goes down or
> causes deadlocks and we are willing to lose as many messages as it takes to
> allow for this.
>
> --
> View this message in context:
> http://www.nabble.com/Options-for-preferring-stability-over-reliability-tp22852174p22852174.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>
--
http://blog.garytully.com
Open Source SOA
http://FUSESource.com
Re: Options for preferring stability over reliability
Posted by marc2112 <ma...@yahoo.com>.
Hey There-
I'm not gonna be of much help off the bat but can certainly commiserate with you. I'm trying to achieve the same goal with similar message loads and am getting frustrated too.
Are you guys running multiple brokers using the network-of-brokers configuration? What I'm seeing right now is roughly once/day the two brokers in my cluster disconnect from each other and then reconnect to each other. It's disruptive to the whole system.
I'm happy to trade off message loss for increased performance and reliability too.
---Marc
________________________________
From: MassDosage <ma...@gmail.com>
To: users@activemq.apache.org
Sent: Thursday, April 2, 2009 1:39:59 PM
Subject: Options for preferring stability over reliability
We are using ActiveMQ quite extensively at Last.fm and are experiencing a
number of issues related to the huge load we are putting on it (at peak
times up to thousands of messages per second). We are using it in a number
of different scenarios but our general use case is that we would much rather
lose hundreds of messages than have deadlocks, out of memory errors and
crashes.
We have turned off producer flow control, set persistence to false, are
using async send, short expiry times on messages etc. Even with this setup
we still regularly (several times per week) run into instances where
ActiveMQ completely fails us - ranging from Out of Memory Exceptions in the
server to blocked Senders to the entire system locking up/freezing with
nothing useful in the log files. These situations force us to restart the
ActiveMQ server regularly and have brought the affected part of our service
to their knees. Ideally this would never happen and in times of heavy load
Senders would not block but instead would drop messages and the server would
do the same instead of running out of memory. Are we missing some
configuration/API options we can use to ensure this behaviour? Does anyone
have any suggestions? I am happy to post our activemq.xml and code samples
if necessary.
I can understand that many of your use cases require reliability and that
you have built much of the system to ensure as few messages get dropped as
possible but we really want the opposite - a system that never goes down or
causes deadlocks and we are willing to lose as many messages as it takes to
allow for this.
--
View this message in context: http://www.nabble.com/Options-for-preferring-stability-over-reliability-tp22852174p22852174.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.