activemq hangups

[Christopher Mihaly <cm...@hotmail.com>]

 I have had two hangup on the message server.  The first happened yesterday. 
After about a week of running smoothly, all the connections to a topic hung. 
All message consumers would hang and all publishers would hang.  I tried
shutting down the server and starting back up, and the first time a client
would come it, it would hang. Subsequent calls would fail with an already
connected error.   Finally, I just destroyed the topic and recreated the
topic and things started working again.   I am afraid it is going to happen
again.  I just do not know why it is hanging.  There was nothing in the
output logs anywhere.  I have a few questions I am hoping someone can
answer:
   1). When I brosed the topic itself, it looks like every message is in
there.   There were a lot of messages.  Is there a limit to how many
messages can be in a topic?  I am assuming there is no limit since we can
have a durable subscriber that is run infrequently and messages can linger
for a while.
    2). That said, some of those message should have been cleaned up because
all subscribers had read them.  When do messages get cleaned out and how? 
Do I have to set something up to do this?
    3). Is there something I can set, a debuging state, that might tell me
what is happening when it gets stuck?  Is there something I can do the next
time it gets stuck to figure out why.  Right now, this is a problem because
things break when this happens, I miss messages and then we loose message
when I destroy and create messages.
 
    I have written another stress case example.   It is just creating
thousands of events by connecting, creating a session, creating a publisher,
publishing a message, and then closing publisher, session and connection. 
It does this for each message, one at a time.   The first time I ran this,
it hung after writting event 1001.  The connection for event 1002 hung
around, the client returned and tried to connect for event 1002 and got a
refused, already connected error.   It repeated this for each one.    I
killed the client, and restarted it, but unlike the above problem, it has
not created another 30000 events and is continuting without hanging.  Again,
I have no idea why it hung on the first run and I have no diagnotic to help
me figure out what happened.  Also, all the events are in the topic even
though there is no subscriber nor durable subscribers on that topic.  When
will these message disappear?

   Any help would be appreciated.
    Chris

-- 
View this message in context: http://www.nabble.com/activemq-hangups-tf1961977.html#a5382883
Sent from the ActiveMQ - User forum at Nabble.com.

Re: activemq hangups

[James Strachan <ja...@gmail.com>]

Hangs can happen when using non-persistent messaging; its a common
problem with having slow consumers....
http://incubator.apache.org/activemq/slow-consumer-handling.html

another workaround is just to increase the usageManager's RAM pool -
see the <usageManager> setting in the example XML
http://incubator.apache.org/activemq/xml-configuration.html

more inline...


On 7/18/06, Christopher Mihaly <cm...@hotmail.com> wrote:
>    1). When I brosed the topic itself, it looks like every message is in
> there.   There were a lot of messages.  Is there a limit to how many
> messages can be in a topic?

No - though there is a RAM limit for non-persistent queues and topics

>     2). That said, some of those message should have been cleaned up because
> all subscribers had read them.  When do messages get cleaned out and how?
> Do I have to set something up to do this?

No it should just happen - its quite asynchronous though due to the
delay in checkpointing from the journal to JDBC and then the timer
fires to clear down any durable topics which have been acknowledged by
all interested subscribers.

I suspect the lockups are more likely to be the non-durable subscribers.


>     3). Is there something I can set, a debuging state, that might tell me
> what is happening when it gets stuck?  Is there something I can do the next
> time it gets stuck to figure out why.  Right now, this is a problem because
> things break when this happens, I miss messages and then we loose message
> when I destroy and create messages.

The best thing is to watch the state of the system with JMX...
http://incubator.apache.org/activemq/jmx.html

But I'd start off by adding a zero or two to the usageManager's RAM
buffer (and having a fairly big heap).

-- 

James
-------
http://radio.weblogs.com/0112098/