You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Paolo28 <pa...@gmail.com> on 2014/05/21 23:21:10 UTC
Camel JMS - memory leaking with IBM MQ
Hi all
In my simple application I have following flow:
1. connect to MQ Queue Manager
2. consume messages via jms component in multithreaded mode (5-8 threads)
AND transacted mode
I´m using standard external Spring JmsTransactionManager which operates
on MQQueueConnectionFactory.
3. proces the messages (simple processor) and store in message in DB
The problem is that there is for sure a memory leak which quite rapidly
consumes memory and in several hours the application crashes with
OutOfMemoryError. The Spring bean definition and wiring is pretty standard
(for example it is inline with the chapter on transactions in the new Camel
book). I made a heap dump and when I inspected that in JVVM there is
suspiciously large amount of instances of com.ibm.* classes. The
MQQueueConnectionFactory which I explicitly define in my Spring file is
there only once as expected but then there are thousands and still growing
counts of following instances which I don´t really understand:
com.ibm.mq.MQClientManagedConnectionFactoryJ11
com.ibm.mq.ClientConnectionRequestInfo
com.ibm.mq.ManagedConnectionStore$Tuple
...
+ hundreds of instances of:
com.ibm.mq.MQManagedConnectionJ11
com.ibm.mq.MQQueueManager
com.ibm.mq.MQQueueSession (same count as previous MQQueueManager)
...
These huge and growing counts are most likely source of the memory leak.
However what I would expect is that Camel just asks the connection factory
for pooled connection (there should be 5-8 connections only) and using
connections it creates Session/Queue etc. objects (these can be also cached
as I understood, I tried to set property cacheLevelName to CACHE_CONSUMER)
and also other needed classes might be created by the MQ impl. but all
should be eligible to garbage collector. Once message processed the
transaction is commited and connection is returned to the pool.
I also assumed the resource management is fully Camel driven including
resource cleanup. The same IBM MQ setup is used in different apps which run
on Spring only and no memory leaks are reported. Of course I don´t want to
blame Camel but currently I´m stuck as I don´t understand internals of this
3rd party code.
Can anyone recommend some steps to reveal the problem? Any help welcome.
Thanks
Paolo
Versions:
Spring: 3.1.4
Camel: 2.11.1
MQ libs: 6.0.2.2
http://camel.apache.org/jms.html
--
View this message in context: http://camel.465427.n5.nabble.com/Camel-JMS-memory-leaking-with-IBM-MQ-tp5751442.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Camel JMS - memory leaking with IBM MQ
Posted by Paolo28 <pa...@gmail.com>.
Yes but for me resource pooling/caching and resource cleanup are two
different things. If I did not use Camel but plain Java I would not face
this issue even with MQQueueConnectionFactory (no wrappers) ...
--
View this message in context: http://camel.465427.n5.nabble.com/Camel-JMS-memory-leaking-with-IBM-MQ-tp5751442p5751779.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Camel JMS - memory leaking with IBM MQ
Posted by Minh Tran <da...@gmail.com>.
Hi
My understanding is that the JMS connection factories were built for use in a J2EE container and if you are using them outside of one, which sounds like you are, then it must be wrapped in either Spring's CachingConnectionFactory or ActiveMQ's PooledConnectionFactory. Though I have read in places that if you're purely consuming from JMS then it's not strictly needed, I haven't tested this myself so don't know if it's true. However there is no harm wrapping it up regardless.
On 27/05/2014, at 7:40 AM, Paolo28 <pa...@gmail.com> wrote:
> Conclusion:
>
> I have solved the issue few days ago by using Spring
> CachingConnectionFactory which wraps MQQueueConnectionFactory - I was aware
> of this possibility but I could not believe it can create memory leak. I
> still don´t understand it and IMO this is bug (MQ?).
>
>
>
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Camel-JMS-memory-leaking-with-IBM-MQ-tp5751442p5751590.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Camel JMS - memory leaking with IBM MQ
Posted by Paolo28 <pa...@gmail.com>.
Conclusion:
I have solved the issue few days ago by using Spring
CachingConnectionFactory which wraps MQQueueConnectionFactory - I was aware
of this possibility but I could not believe it can create memory leak. I
still don´t understand it and IMO this is bug (MQ?).
--
View this message in context: http://camel.465427.n5.nabble.com/Camel-JMS-memory-leaking-with-IBM-MQ-tp5751442p5751590.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Camel JMS - memory leaking with IBM MQ
Posted by Paolo28 <pa...@gmail.com>.
Hi,
I use IBM MQ not ActiveMQ but I considered wrapping MQ conn. factory with
Spring impl. but anyway caching is always about performance boosts so the
calling code does not create expensive objects over and over if not needed.
However if not caching it should not result in memory leak. Resoruces need
to be cleaned up anyway whether caching or not or returned to pool where
there is only configured amount of resources kept.
--
View this message in context: http://camel.465427.n5.nabble.com/Camel-JMS-memory-leaking-with-IBM-MQ-tp5751442p5751451.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Camel JMS - memory leaking with IBM MQ
Posted by Willem Jiang <wi...@gmail.com>.
Did you use cached or pooled connection factory just as this[1] suggested ?
[1]http://activemq.apache.org/jmstemplate-gotchas.html
--
Willem Jiang
Red Hat, Inc.
Web: http://www.redhat.com
Blog: http://willemjiang.blogspot.com (English)
http://jnn.iteye.com (Chinese)
Twitter: willemjiang
Weibo: 姜宁willem
On May 22, 2014 at 5:21:37 AM, Paolo28 (paolo.sagat@gmail.com) wrote:
> Hi all
> In my simple application I have following flow:
>
> 1. connect to MQ Queue Manager
> 2. consume messages via jms component in multithreaded mode (5-8 threads)
> AND transacted mode
> I´m using standard external Spring JmsTransactionManager which operates
> on MQQueueConnectionFactory.
>
> 3. proces the messages (simple processor) and store in message in DB
>
> The problem is that there is for sure a memory leak which quite rapidly
> consumes memory and in several hours the application crashes with
> OutOfMemoryError. The Spring bean definition and wiring is pretty standard
> (for example it is inline with the chapter on transactions in the new Camel
> book). I made a heap dump and when I inspected that in JVVM there is
> suspiciously large amount of instances of com.ibm.* classes. The
> MQQueueConnectionFactory which I explicitly define in my Spring file is
> there only once as expected but then there are thousands and still growing
> counts of following instances which I don´t really understand:
>
> com.ibm.mq.MQClientManagedConnectionFactoryJ11
> com.ibm.mq.ClientConnectionRequestInfo
> com.ibm.mq.ManagedConnectionStore$Tuple
> ...
>
> + hundreds of instances of:
> com.ibm.mq.MQManagedConnectionJ11
> com.ibm.mq.MQQueueManager
> com.ibm.mq.MQQueueSession (same count as previous MQQueueManager)
> ...
>
>
> These huge and growing counts are most likely source of the memory leak.
> However what I would expect is that Camel just asks the connection factory
> for pooled connection (there should be 5-8 connections only) and using
> connections it creates Session/Queue etc. objects (these can be also cached
> as I understood, I tried to set property cacheLevelName to CACHE_CONSUMER)
> and also other needed classes might be created by the MQ impl. but all
> should be eligible to garbage collector. Once message processed the
> transaction is commited and connection is returned to the pool.
>
> I also assumed the resource management is fully Camel driven including
> resource cleanup. The same IBM MQ setup is used in different apps which run
> on Spring only and no memory leaks are reported. Of course I don´t want to
> blame Camel but currently I´m stuck as I don´t understand internals of this
> 3rd party code.
>
> Can anyone recommend some steps to reveal the problem? Any help welcome.
>
> Thanks
> Paolo
>
>
> Versions:
>
> Spring: 3.1.4
> Camel: 2.11.1
> MQ libs: 6.0.2.2
>
> http://camel.apache.org/jms.html
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Camel-JMS-memory-leaking-with-IBM-MQ-tp5751442.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>