memory use, problems with stomp

[snacktime <sn...@gmail.com>]

Using the latest RC I have a test script that sends about 10,000 short
messages to a queue and then reads them.  Doing that a couple of times makes
activemq go from using up 48mb ram to 152 mb.  Also, after a while I can't
read the messages that are in the queue, and I get the following message at
the console.  Maybe I'm just unlucky, but activemq breaks every time I try
to use it.  Before it was a bug with stomp that was supposedly fixed in the
latest RC.

Failed to unregister mbean:
org.apache.activemq:BrokerName=localhost,Type=Connection,ConnectorName=stomp,Connection=ID_myhost-59310-1160695521231-3_5
-- 
View this message in context: http://www.nabble.com/memory-use%2C-problems-with-stomp-tf2434037.html#a6787277
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: memory use, problems with stomp

[snacktime <sn...@gmail.com>]

Some more information on the problems I've been seeing with stomp.   When
sending messages to a queue, activemq still thinks there is a consumer
connected even after it is long gone, and subscribers are then unable to
read messages from the queue.  I've seen phantom consumers consuming
messages, and queue counts jump from the thousands to -1, -2 for no apparent
reason.  I'll often connect to a queue and try to read messages, and JMX
will show the queue count decrement by one even though the client doesn't
get anything, and then the client will just sit there forever with jmx
showing thousands of messages in the queue.  To make matters worse there
appears to be no way to fix a queue once it's gotten into a bad state. 
Restarting activemq will get rid of the phantom consumer but it's impossible
to read the messages from the queue.  

I'm also seeing something strange which could very well be my
misunderstanding of how activemq works.  I send 5000 messages to an empty
queue then try to purge the queue in JMX with throws an exception.  JMX then
shows 4000 messages in the queue (what happened to the other 1000?), and
when I consume the messages I get all 5000 back out, but JMX still shows one
message in the queue.   And sometimes purge has no effect at all, while
resetstatistics clears out the queue??

It's all so strange I really have no idea how to debug it, and I'm not a
java developer.  
-- 
View this message in context: http://www.nabble.com/memory-use%2C-problems-with-stomp-tf2434037.html#a6803253
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: memory use, problems with stomp

[snacktime <sn...@gmail.com>]

Ok I'm pretty sure this particular problem is with the ruby stomp
client.  After writing a simple stomp server in ruby last night I
found out that the ruby client will not read the end of frame null
character unless it's followed by a newline, which probably accounts
for some of the behavior I'm seeing with activemq.

There is still the issue of activemq not unregistering consumers after
they have disconnected, but hopefully that's actually the only real
bug in activemq itself.  I'll confirm the above after I get some time
to test it.

Re: memory use, problems with stomp

[snacktime <sn...@gmail.com>]

Also, after getting in a hung state like that, restarting activemq
always shows one message in transaction(s) was recovered.  Restarting
does not let the remaining messages be read either, and trying to read
them results in the same pattern as when the server first hung.  JMX
shows one transaction read if you try to read from the queue, client
gets nothing, and when client disconnects JMX still shows an active
consumer.

Chris

Re: memory use, problems with stomp

[snacktime <sn...@gmail.com>]

> By 'break' do you mean getting an OutOfMemoryException? If so try
> setting your heap to something large like -Xmx512m - Derby tends to
> hog a fair bit of RAM. If you don't want to run with a large heap try
> switching to mysql or postgresql.

No, I don't get any error messages at all.  Here is what I do.

-  send 20,000 messages into the queue and disconnect
- connect/subscribe to the queue and read the messages back out
- repeat

After 4-6 cycles of this the messages just stop coming out of the
queue.  For example I'll be reading the queue which has 15,000
messages in it, and it will just stop after reading 5000.  If I
disconnect JMX still shows a consumer.  If I connect again JMX shows
one message consumed, but the client never gets it, and if I
disconnect JMX now shows 2 consumers still connected, and so on.
Every time I connect after this it's the same pattern.  JMX shows one
message consumed, client gets nothing, and another consumer is shown
in JMX.

During this time activemq goes from around 50mb resident ram to around
160.  There is still free memory and no exceptions or errors.

I just repeated this whole process again while writing this.

Chris

Re: memory use, problems with stomp

[James Strachan <ja...@gmail.com>]

On 10/16/06, snacktime <sn...@gmail.com> wrote:
> On 10/16/06, James Strachan <ja...@gmail.com> wrote:
> > On 10/13/06, snacktime <sn...@gmail.com> wrote:
> > > Ok some more information on the problems with stomp.  I have a test script
> > > that sends 10000 messages to a queue and then another that reads them.
> >
> > Are you using persistence?
>
> Yes.   With the default derby db.

By 'break' do you mean getting an OutOfMemoryException? If so try
setting your heap to something large like -Xmx512m - Derby tends to
hog a fair bit of RAM. If you don't want to run with a large heap try
switching to mysql or postgresql.

-- 

James
-------
http://radio.weblogs.com/0112098/

Re: memory use, problems with stomp

[snacktime <sn...@gmail.com>]

On 10/16/06, James Strachan <ja...@gmail.com> wrote:
> On 10/13/06, snacktime <sn...@gmail.com> wrote:
> > Ok some more information on the problems with stomp.  I have a test script
> > that sends 10000 messages to a queue and then another that reads them.
>
> Are you using persistence?

Yes.   With the default derby db.

Chris

Re: memory use, problems with stomp

[James Strachan <ja...@gmail.com>]

On 10/13/06, snacktime <sn...@gmail.com> wrote:
> Ok some more information on the problems with stomp.  I have a test script
> that sends 10000 messages to a queue and then another that reads them.

Are you using persistence?

> After a couple of iterations I get a phantom consumer and then I can't read
> anything.  Trying to purge the queue via JMS results in a
> java.util.CurrentModificationException.

I think we've fixed that one fairly recently. FWIW in future if you
hit any exception, sending the full stack trace helps us fix any
issues.

-- 

James
-------
http://radio.weblogs.com/0112098/

Re: memory use, problems with stomp

[snacktime <sn...@gmail.com>]

Ok some more information on the problems with stomp.  I have a test script
that sends 10000 messages to a queue and then another that reads them. 
After a couple of iterations I get a phantom consumer and then I can't read
anything.  Trying to purge the queue via JMS results in a
java.util.CurrentModificationException.  After a few moments I get the
following message from activemq and the phantom consumer disappears and the
queue size in JMX shows -1.
Still can't purge the queue because of the above error.  If I restart the
broker it say's it recovered one transaction, but the queue is completely
gone.

Failed to unregister mbean:
org.apache.activemq:BrokerName=catalog1,Type=Connection,ConnectorName=stomp,Connection=ID_catalog1.paymentonline.net-56165-1160717547729-3_15

-- 
View this message in context: http://www.nabble.com/memory-use%2C-problems-with-stomp-tf2434037.html#a6790388
Sent from the ActiveMQ - User mailing list archive at Nabble.com.