Memory leaks in AMQ 5.0?

[Arjen van der Meijden <ac...@tweakers.net>]

Hi list,

Currently we're seeing some strange behaviour from ActiveMQ 5.0 (release 
version).

In our set-up we have one ActiveMQ instance with four queues on it and 
all communication with AMQ is via the Stomp-interface. Although we 
cleaned the activemq.xml a bit, we basically left all settings to their 
respective defaults. We did change the Xmx-parameter in the bin-file to 
2048M though since the below described problems are much worse with only 
the default 512M.

We have five apache/php-webservers that connect to AMQ and produce one 
or two message on every page request. They use the php-implementation of 
Stomp and about 70% of those messages are persistent.
In peak moments, we do about 70-100 pageviews/second (and thus as many 
connections to AMQ), which boils down to per-queue peaks of somewhere in 
the 50 messages/second area.

Each queue has only one consumer. Those consumers throttle their 
consumptionspeed based on the systemload, but normally they consume each 
message as it enters a queue.
The connection/subscription from the consumers to ActiveMQ is opened at 
start and then kept open for the duration of the program.

Under normal conditions there is at most one message in a queue waiting 
to be consumed, but we've also seen it handle a (growing) queue of 100k 
messages with ease.

We're noticing decreasing performance and increasing memory usage over 
time. So two days ago I started logging the CC:PrintGCDetails and made a 
graph of the free new/total heap before and after each young-gc (which 
also shows the full gc's):
http://achelois.tweakers.net/~acm/tnet/activemq-gc.png

As you can see the new-part of the heap (green lines) isn't really 
moving much, but the total heap (orange/red lines) keeps growing during 
the day. What you also can see in the graph is the two nights AMQ was 
running, and where no increase happened (from 20k to 40k seconds and 
100k to 120k). Our nights are, obviously, much quieter than our days.

I've tried the Concurrent GC, but reverted to the Parallel GC since the 
first one couldn't keep up when the heap grew over 700MB. AMQ is running 
on a dual processor opteron with 8GB of memory and several 15k-rpm scsi 
disks in a raid 5 for its storage.

I'm clueless as to what this increase in memory consumption may cause 
and/or how to prevent it.

Any ideas?


Best regards,

Arjen van der Meijden
Tweakers.net

Re: Memory leaks in AMQ 5.0?

[Arjen van der Meijden <ac...@tweakers.net>]

Is there noone with an idea?

I've made a new graph with a longer timeperiod:
http://achelois.tweakers.net/~acm/tnet/long-activemq-gc.png

For now we can only fix this by periodically restart ActiveMQ, but 
that's not a very nice aproach.

Best regards,

Arjen

Arjen van der Meijden wrote:
> Hi list,
> 
> Currently we're seeing some strange behaviour from ActiveMQ 5.0 (release 
> version).
> 
> In our set-up we have one ActiveMQ instance with four queues on it and 
> all communication with AMQ is via the Stomp-interface. Although we 
> cleaned the activemq.xml a bit, we basically left all settings to their 
> respective defaults. We did change the Xmx-parameter in the bin-file to 
> 2048M though since the below described problems are much worse with only 
> the default 512M.
> 
> We have five apache/php-webservers that connect to AMQ and produce one 
> or two message on every page request. They use the php-implementation of 
> Stomp and about 70% of those messages are persistent.
> In peak moments, we do about 70-100 pageviews/second (and thus as many 
> connections to AMQ), which boils down to per-queue peaks of somewhere in 
> the 50 messages/second area.
> 
> Each queue has only one consumer. Those consumers throttle their 
> consumptionspeed based on the systemload, but normally they consume each 
> message as it enters a queue.
> The connection/subscription from the consumers to ActiveMQ is opened at 
> start and then kept open for the duration of the program.
> 
> Under normal conditions there is at most one message in a queue waiting 
> to be consumed, but we've also seen it handle a (growing) queue of 100k 
> messages with ease.
> 
> We're noticing decreasing performance and increasing memory usage over 
> time. So two days ago I started logging the CC:PrintGCDetails and made a 
> graph of the free new/total heap before and after each young-gc (which 
> also shows the full gc's):
> http://achelois.tweakers.net/~acm/tnet/activemq-gc.png
> 
> As you can see the new-part of the heap (green lines) isn't really 
> moving much, but the total heap (orange/red lines) keeps growing during 
> the day. What you also can see in the graph is the two nights AMQ was 
> running, and where no increase happened (from 20k to 40k seconds and 
> 100k to 120k). Our nights are, obviously, much quieter than our days.
> 
> I've tried the Concurrent GC, but reverted to the Parallel GC since the 
> first one couldn't keep up when the heap grew over 700MB. AMQ is running 
> on a dual processor opteron with 8GB of memory and several 15k-rpm scsi 
> disks in a raid 5 for its storage.
> 
> I'm clueless as to what this increase in memory consumption may cause 
> and/or how to prevent it.
> 
> Any ideas?
> 
> 
> Best regards,
> 
> Arjen van der Meijden
> Tweakers.net
>