Re: Options for preferring stability over reliability

[geoffmciver <ge...@team.telstraclear.co.nz>]

I too am seeing an issue that activemq freezes completely, Im not sure if it
is under "heavy" load at the time.

I'm using it as a simple queue with one producer and one consumer.

Nothing appears in the logs and wrappers reports that it "couldn't ping JVM
for xx seconds" and restarts ActiveMQ.  But sometimes wrappers fails to
restart it and it just sits in a totally hung state.

There are no JVM crash logs or anything in the activemq.log that reports
anything.  Its just like it freezes in mid air.

I wouldn't imagine that the load is high as we one have 100's of messages a
second and not 1000's as quoted by many other ActiveMQ users.

With our usage profile it appears to run for 1 week and then crash...   It
smealt like a memory leak issue but I would have expected the JVM to report
the fact or have Heap space errors appearing some where.

is there any way to get more verbose logging or would you recommend another
course of action.

I don't want to hijack this thread, I just think that my issue could be the
same.


Thanks

Geoff McIver







MassDosage wrote:
> 
> We are using ActiveMQ quite extensively at Last.fm and are experiencing a
> number of issues related to the huge load we are putting on it (at peak
> times up to thousands of messages per second). We are using it in a number
> of different scenarios but our general use case is that we would much
> rather lose hundreds of messages than have deadlocks, out of memory errors
> and crashes. 
> 
> We have turned off producer flow control, set persistence to false, are
> using async send, short expiry times on messages etc. Even with this setup
> we still regularly (several times per week) run into instances where
> ActiveMQ completely fails us - ranging from Out of Memory Exceptions in
> the server to blocked Senders to the entire system locking up/freezing
> with nothing useful in the log files. These situations force us to restart
> the ActiveMQ server regularly and have brought the affected part of our
> service to their knees. Ideally this would never happen and in times of
> heavy load Senders would not block but instead would drop messages and the
> server would do the same instead of running out of memory. Are we missing
> some configuration/API options we can use to ensure this behaviour? Does
> anyone have any suggestions? I am happy to post our activemq.xml and code
> samples if necessary.
> 
> I can understand that many of your use cases require reliability and that
> you have built much of the system to ensure as few messages get dropped as
> possible but we really want the opposite - a system that never goes down
> or causes deadlocks and we are willing to lose as many messages as it
> takes to allow for this. 
> 
> 

-- 
View this message in context: http://www.nabble.com/Options-for-preferring-stability-over-reliability-tp22852174p23360873.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Options for preferring stability over reliability

[geoffmciver <ge...@team.telstraclear.co.nz>]

MassDosage wrote:
> 
> What is this wrapper you are talking about?
> 
> I've never seen that "couldn't ping JVM" message so I'm not sure whether
> this is the same issue or not.
> 
> 
> 

ActiveMQ gets "shipped" with Java wrappers http://wrapper.tanukisoftware.org

its a way of daemonising java apps.  It forks the JVM process as its child
and keeps tabs on it.  If it dies it can restart it.   

the underlying issue is the same I think as all wrappers does it restart
activemq.  it I run it without wrappers the result is the same as you...  
the JVM just hangs with no errors to the logs or console.

-- 
View this message in context: http://www.nabble.com/Options-for-preferring-stability-over-reliability-tp22852174p23395150.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Options for preferring stability over reliability

[MassDosage <ma...@gmail.com>]

What is this wrapper you are talking about?

I've never seen that "couldn't ping JVM" message so I'm not sure whether
this is the same issue or not.


-- 
View this message in context: http://www.nabble.com/Options-for-preferring-stability-over-reliability-tp22852174p23384104.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.