You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Simon-Pierre LeBel <sp...@iweb.com> on 2011/07/13 15:56:52 UTC
Possible memory leak
Hi,
I have the following setup:
- ActiveMQ 5.3.2
- Pure Master/Slave
- ~800 consumers
- ~320 queues
- ~650 topics
My monitoring shows that the memory usage is going up daily, in 5 days
the heap used went up by 50MB without any changes in consumers, queues
or topics count. I tried to force GC but the memory usage doesn't go
down which is probably caused by a memory leak.
I've attached screenshots of heap profiling done with MAT, the first one
was done on monday, the second today. The biggest difference between the
two days is org.apache.activemq.openwire.OpenWireFormat memory usage.
Can you help me identify the cause and fix this issue ?
Let me know if you need more informations.
Regards,
--
Simon-Pierre LeBel [iWeb]
IT Architecture Specialist / Technical Lead
Spécialiste de l'Architecture TI / Leader Technique
http://www.iWeb.com/
Re: InactivityMonitor - OOME on sleep
Posted by "tabish121@gmail.com" <ta...@gmail.com>.
I think a quick fix is to change from calling scheduleAtFixedRate to just
calling schedule, this different being the way the interval between
executions is controlled.
For scheduleAtFixedRate which we use now execution rate is scheduled based
on the time that the task was first scheduled.
> In fixed-rate execution, each execution is scheduled relative to the
> scheduled execution time of the initial execution. If an execution is
> delayed for any reason (such as garbage collection or other background
> activity), two or more executions will occur in rapid succession to "catch
> up." In the long run, the frequency of execution will be exactly the
> reciprocal of the specified period (assuming the system clock underlying
> Object.wait(long) is accurate).
>
For schedule the rate is controlled based on the time of the last execution:
> In fixed-delay execution, each execution is scheduled relative to the
> actual execution time of the previous execution. If an execution is
> delayed for any reason (such as garbage collection or other background
> activity), subsequent executions will be delayed as well. In the long run,
> the frequency of execution will generally be slightly lower than the
> reciprocal of the specified period (assuming the system clock underlying
> Object.wait(long) is accurate).
>
In the case of the write check task I'd think that it'd be better to have
the occasional delays that can occur because of garbage collection etc vs
having write checks occur in quick rapid fire succession because things got
slowed down since once you do one write check, then you don't need to do
another until the next configured interval.
Regards
Tim.
http://fusesource.com
Gary Tully wrote:
>
> hmm, that sounds like a reasonable theory... seems like a scheduled
> executor is the answer or a scheduler that uses cpu relative time...
> so that sleep time is ignored.
>
> Can you raise a jira issue to track this, there are a few schedulers
> in the code base, so they may all need the same treatment.
>
> On 13 July 2011 22:45, Michael Brewer-Davis
> <michael@tech4learning.com> wrote:
>> Using activemq 5.4.0 for a P2P desktop applicaiton, I get an OOME when my
>> computer awakes from sleep:
>>
>> Exception in thread "InactivityMonitor WriteCheck"
>> java.lang.OutOfMemoryError: unable to create new native thread
>>
>> The cause appears to be:
>> - WRITE_CHECK_TIMER schedules checks at a fixed rate
>> - ASYNC_THREADS uses an unbounded thread pool to service the checks
>>
>> When the computer wakes, WRITE_CHECK_TIMER satuates the system with pent
>> up
>> requests and uses up all available threads.
>>
>> Am I right in thinking:
>> - this is an issue
>> - some options for resolution are:
>> bounding the ASYNC_THREADS
>> converting WRITE_CHECK_TIMER to a scheduled executor, which would run
>> the submitted task serially
>>
>>
>> michael
>>
>>
>
>
>
> --
> http://fusesource.com
> http://blog.garytully.com
>
--
View this message in context: http://activemq.2283324.n4.nabble.com/Possible-memory-leak-tp3665094p3667681.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: InactivityMonitor - OOME on sleep
Posted by Gary Tully <ga...@gmail.com>.
hmm, that sounds like a reasonable theory... seems like a scheduled
executor is the answer or a scheduler that uses cpu relative time...
so that sleep time is ignored.
Can you raise a jira issue to track this, there are a few schedulers
in the code base, so they may all need the same treatment.
On 13 July 2011 22:45, Michael Brewer-Davis <mi...@tech4learning.com> wrote:
> Using activemq 5.4.0 for a P2P desktop applicaiton, I get an OOME when my
> computer awakes from sleep:
>
> Exception in thread "InactivityMonitor WriteCheck"
> java.lang.OutOfMemoryError: unable to create new native thread
>
> The cause appears to be:
> - WRITE_CHECK_TIMER schedules checks at a fixed rate
> - ASYNC_THREADS uses an unbounded thread pool to service the checks
>
> When the computer wakes, WRITE_CHECK_TIMER satuates the system with pent up
> requests and uses up all available threads.
>
> Am I right in thinking:
> - this is an issue
> - some options for resolution are:
> bounding the ASYNC_THREADS
> converting WRITE_CHECK_TIMER to a scheduled executor, which would run
> the submitted task serially
>
>
> michael
>
>
--
http://fusesource.com
http://blog.garytully.com
InactivityMonitor - OOME on sleep
Posted by Michael Brewer-Davis <mi...@tech4learning.com>.
Using activemq 5.4.0 for a P2P desktop applicaiton, I get an OOME when
my computer awakes from sleep:
Exception in thread "InactivityMonitor WriteCheck"
java.lang.OutOfMemoryError: unable to create new native thread
The cause appears to be:
- WRITE_CHECK_TIMER schedules checks at a fixed rate
- ASYNC_THREADS uses an unbounded thread pool to service the checks
When the computer wakes, WRITE_CHECK_TIMER satuates the system with pent
up requests and uses up all available threads.
Am I right in thinking:
- this is an issue
- some options for resolution are:
bounding the ASYNC_THREADS
converting WRITE_CHECK_TIMER to a scheduled executor, which would
run the submitted task serially
michael
Re: Possible memory leak
Posted by Dejan Bosanac <de...@nighttale.net>.
Please raise a Jira for it and attach a heap dump there.
Regards
--
Dejan Bosanac - http://twitter.com/dejanb
-----------------
The experts in open source integration and messaging - http://fusesource.com
ActiveMQ in Action - http://www.manning.com/snyder/
Blog - http://www.nighttale.net
On Fri, Jul 15, 2011 at 1:56 PM, Simon-Pierre LeBel <sp...@iweb.com>wrote:
>
> Le 11-07-13 10:11, Dejan Bosanac a écrit :
> > can you test the latest release 5.5.0 (or the latest 5.6-SNAPSHOT) and
> > confirm that problem still exists?
>
> I did some tests with 5.5.0 and it still exists. Heavy usage of JMX
> seems to be the source of the problem. The test I ran consisted of a
> nagios check that uses JMX to verify consumer count, persistence and
> heap/non-heap usage.
>
> Let me know if you need a heap dump or anything else.
>
> Regards,
>
> --
> Simon-Pierre LeBel [iWeb]
> IT Architecture Specialist / Technical Lead
> Spécialiste de l'Architecture TI / Leader Technique
> http://www.iWeb.com/
>
>
>
Re: Possible memory leak
Posted by Simon-Pierre LeBel <sp...@iweb.com>.
Le 11-07-13 10:11, Dejan Bosanac a écrit :
> can you test the latest release 5.5.0 (or the latest 5.6-SNAPSHOT) and
> confirm that problem still exists?
I did some tests with 5.5.0 and it still exists. Heavy usage of JMX
seems to be the source of the problem. The test I ran consisted of a
nagios check that uses JMX to verify consumer count, persistence and
heap/non-heap usage.
Let me know if you need a heap dump or anything else.
Regards,
--
Simon-Pierre LeBel [iWeb]
IT Architecture Specialist / Technical Lead
Spécialiste de l'Architecture TI / Leader Technique
http://www.iWeb.com/
Re: Possible memory leak
Posted by Dejan Bosanac <de...@nighttale.net>.
Hi,
can you test the latest release 5.5.0 (or the latest 5.6-SNAPSHOT) and
confirm that problem still exists?
Regards
--
Dejan Bosanac - http://twitter.com/dejanb
-----------------
The experts in open source integration and messaging - http://fusesource.com
ActiveMQ in Action - http://www.manning.com/snyder/
Blog - http://www.nighttale.net
On Wed, Jul 13, 2011 at 3:56 PM, Simon-Pierre LeBel <sp...@iweb.com>wrote:
> Hi,
>
> I have the following setup:
> - ActiveMQ 5.3.2
> - Pure Master/Slave
> - ~800 consumers
> - ~320 queues
> - ~650 topics
>
> My monitoring shows that the memory usage is going up daily, in 5 days
> the heap used went up by 50MB without any changes in consumers, queues
> or topics count. I tried to force GC but the memory usage doesn't go
> down which is probably caused by a memory leak.
>
> I've attached screenshots of heap profiling done with MAT, the first one
> was done on monday, the second today. The biggest difference between the
> two days is org.apache.activemq.openwire.OpenWireFormat memory usage.
>
> Can you help me identify the cause and fix this issue ?
>
> Let me know if you need more informations.
>
> Regards,
>
> --
> Simon-Pierre LeBel [iWeb]
> IT Architecture Specialist / Technical Lead
> Spécialiste de l'Architecture TI / Leader Technique
> http://www.iWeb.com/
>
>