You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by ee7arh <an...@2e-systems.com> on 2009/03/18 16:07:40 UTC
References to objects processed in routes held, eventually run out
of memory
Hi,
I have defined a series of routes in Camel which effectively routes messages
from queues, to bean processors and back to queues until at the very end the
message is written to a database and processing is complete.
My application ran out of memory after running for a while and when I
debugged with JMap and JHat I found that every object I created in my bean
processors and routed through Camel still existed in memory. It was as
though nothing ever got garbage collected.
I have no lists or caches in my application.
One thing I noticed from Jhat is that my objects were still referenced by
camel / jms:
References to this object:
org.apache.camel.component.jms.JmsExchange@0x8d877ea0 (48 bytes) : field in
I verified that all my queues are empty so I'm wondering how it can be that
my objects are still referenced by Camel and thus haven't been garbage
collected?
Any theories?
Thanks
Andrew
--
View this message in context: http://www.nabble.com/References-to-objects-processed-in-routes-held%2C-eventually-run-out-of-memory-tp22580791p22580791.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: References to objects processed in routes held, eventually run
out of memory
Posted by Claus Ibsen <cl...@gmail.com>.
On Thu, Mar 19, 2009 at 2:24 PM, ee7arh <an...@2e-systems.com> wrote:
>
> Hi,
>
> Thanks for response.
>
> I'm using Camel 1.6.0 and ActiveMQ 5.2.0. I also had the same results with
> camel 1.5 and activeMQ 5.1
>
> I have attached a graph from Jconsole showing what happens as the
> application starts, runs for a about 1.5 hours then crashes with out of
> memory. http://www.nabble.com/file/p22600122/jconsole-graphs.jpg
>
> Notes:
>
> - I set the maximum memory to 64MB using -Xmx64M -server. normally I would
> use more but then the problem takes longer to reproduce
> - Notice the number of loaded classes never reduces. I also have another
> application which shows the same pattern.
> - As the application starts to run out of memory, the CPU usage goes
> extremely high
>
> Here is my route:
>
> from("jms:queue:mobileExtTriggerInvites")
>
> .errorHandler(deadLetterChannel("seda:errors").maximumRedeliveries(0))
>
> // Exception Handling
> .onException(JAXBException.class)
> .to("jms:queue:unmarshallableRequest").end()
> .onException(UnexpectedEventException.class)
> .to("jms:queue:unexpectedEvent").end()
>
> // Log Incoming XML
> .to("log:incomingEventXmlLogger?level=INFO")
>
> // Attempt to Unmarshall object expected on this queue
> .to("bean:eventMarshaller?methodName=unmarshallTriggerInvites")
>
> // Generate Service Events from this External Event
>
> .to("bean:serviceEventGenerator?methodName=procecssTriggerInvites")
>
> // Split generated Events out of List into individual events
> .splitter(body())
>
> // Destination queue for further processing
> .to("jms:queue:serviceEventQueue");
>
> You can see that I have 2 bean processors, I have also attached the code for
> those.
>
> http://www.nabble.com/file/p22600122/EventMarhaller.txt EventMarhaller.txt
> http://www.nabble.com/file/p22600122/ServiceEventGenerator.txt
> ServiceEventGenerator.txt
>
> Perhaps it's useful, perhaps not but I read a blog article which sounded
> very similar to the problem I am having, link below. Basically it looks like
> Camel or ActiveMQ still has a reference to every single object I create and
> send through the routing - this would surely explain why the number of
> loaded classes gets bigger and bigger the more the application runs.
>
> http://dertompson.com/2007/08/01/fighting-the-outofmemoryerror/
> http://dertompson.com/2007/08/01/fighting-the-outofmemoryerror
>
> using JConsole, I also periodically purged the end queue and this made no
> difference.
>
> If it helps, I am able to provide a heapdump to anyone for analysis with
> jHat. As mentioned in my 1st post, all instances of classes created during
> the application lifecycle continue to live on in memory. It appears that my
> objects are referenced by ActiveMQMessage
>
> For example, I will trace 1 of those classes created in code attached:
>
> Class:
> class
> com.ee.berbe.mobile.servicelogic.flightops.events.service.TriggerMMSInvitations
>
> References to this object:
> org.apache.activemq.command.ActiveMQObjectMessage@0xafab8fd8 (174 bytes) :
> field object
>
> --> -->
>
> Class:
> class org.apache.activemq.command.ActiveMQObjectMessage
>
> References to this object:
> java.util.LinkedHashMap$Entry@0xafab8970 (32 bytes) : field value
> java.util.LinkedHashMap$Entry@0xafab8810 (32 bytes) : field value
I was wondering if those LinkedHashMaps could be a WeakHashMap, on the
ActiveMQObjectMessage, instead?
>
>
> So it seems that ActiveMQ is still storing all my objets on a LinkedHashMap
> but I've no idea why.
>
> Thanks again for any help
>
> Andrew
>
>
> willem.jiang wrote:
>>
>> Hi
>> Which version of Camel are you using ?
>> Can you show us you routing rules and your processor codes?
>>
>> They will help us to dig the issue.
>>
>> Willem
>>
>>
> --
> View this message in context: http://www.nabble.com/References-to-objects-processed-in-routes-held%2C-eventually-run-out-of-memory-tp22580791p22600122.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>
--
Claus Ibsen
Apache Camel Committer
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Re: References to objects processed in routes held, eventually run
out of memory
Posted by ee7arh <an...@2e-systems.com>.
Hi,
My application has continued to run now for almost a full 2 days with 256MB
and I have been connected to it continuously with jconsole via the camel JMX
component.
When I first looked at it this morning my first thought was "oh dear"
because memory heap was hovering dangerously close to the maximum memory
allocated (256MB). However due to a problem with an external connection, the
application had effectively been idle for about 7 hours. When I restored the
connection and the application began catching up on it's processing I was
expecting an iminent "OutOfMemory" error but to my relief, the heap memory
dropped dramatically down from around 250MB to about 45MB!!!
Here are the jconsole graphs to show this:
http://www.nabble.com/file/p22644524/jconsole-graph-3.jpg
However - I still noticed from the jconsole graph that the number of classes
loaded never reduced, always increased just as before. So out of interest I
took a memory dump with JMAP and analysed it again with JHAT. What I found
was encouraging.... Although I cannot explain why there are still 1-2
thousand instances of my objects in memory, the number is definitely a lot
less than the total number of classes loaded since the application started
which proves that they are being garbage collected afterall. :jumping:
So why jconsole does not show this I do not have any idea. Perhaps an issue
with the JMX component? Or perhaps I am mis-interpretting this statistic.
Another question that arises is why ActiveMQ/Camel is holding onto objects
in memory until almost the last moment before garbage collecting. As I found
out when using a lower amount of memory, it can lead to an "OutOfMemory"
error. Perhaps this is a feature or maybe something which needs to be looked
at. What worries me is that I'm not even doing that much processing at the
moment and plan to significantly increase this so hope that I won't need to
add lots more memory. Maybe I also need to consider sending more lightweight
objects through the routing and make more use of the "Claim check" pattern.
Any more thoughts?
Thanks
Andrew
--
View this message in context: http://www.nabble.com/References-to-objects-processed-in-routes-held%2C-eventually-run-out-of-memory-tp22580791p22644524.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: References to objects processed in routes held, eventually run
out of memory
Posted by ee7arh <an...@2e-systems.com>.
Hi again,
I added this component into my camel xml config and added a couple of extra
jar files from activemq to get it working:
<!-- configure the Camel JMS consumer to use the ActiveMQ broker declared
above -->
<bean id="jms"
class="org.apache.activemq.camel.component.ActiveMQComponent">
<property name="brokerURL" value="vm://localhost"/>
</bean>
I continued to use the id="jms" to avoid modifying all my routes.
Unfortunately, it did not make any difference. I left the application
running for about 1 hour now and the classes continue to increase steadily
just as before :-(
Thanks
Andrew
Claus Ibsen-2 wrote:
>
> Hi
>
> You should use the AMQ component instead of the generic JMS component
> http://camel.apache.org/activemq.html
>
>
>
>
>
--
View this message in context: http://www.nabble.com/References-to-objects-processed-in-routes-held%2C-eventually-run-out-of-memory-tp22580791p22619315.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: References to objects processed in routes held, eventually run
out of memory
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
You should use the AMQ component instead of the generic JMS component
http://camel.apache.org/activemq.html
On Fri, Mar 20, 2009 at 11:45 AM, ee7arh <an...@2e-systems.com> wrote:
>
> Hi,
>
> Here it comes:
>
> <!-- A local ativeMQ broker -->
>
> <broker:broker useJmx="true" persistent="false" brokerName="localhost">
> <broker:transportConnectors>
> <broker:transportConnector name="tcp" uri="tcp://localhost:61616"/>
> </broker:transportConnectors>
> </broker:broker>
>
> <!-- Connection factory for ActiveMQ endpoints -->
> <bean id="connectionFactoryActiveMQ"
> class="org.apache.activemq.ActiveMQConnectionFactory">
> <property name="brokerURL" value="vm://localhost"/>
> </bean>
>
> <!-- configure the Camel JMS consumer to use the ActiveMQ broker declared
> above -->
> <bean id="jms" class="org.apache.camel.component.jms.JmsComponent">
> <property name="connectionFactory" ref="connectionFactoryActiveMQ"/>
> </bean>
>
> and my camel context for completeness:
>
> <camel:camelContext id="camel"
> xmlns="http://activemq.apache.org/camel/schema/spring">
> <camel:package>com.ee.berbe.mobile</camel:package>
> <!-- enable JMX connector so we can connect to the server and
> browse mbeans -->
> <camel:jmxAgent id="agent" createConnector="true"
> registryPort="1098"/>
> </camel:camelContext>
>
> here is ActiveMQ starting:
>
> [2009-03-19 11:28:37,785] org.apache.activemq.broker.BrokerService INFO -
> ActiveMQ 5.2.0 JMS Message Broker (localhost) is starting
> [2009-03-19 11:28:37,785] org.apache.activemq.broker.BrokerService INFO -
> For help or more information please see: http://activemq.apache.org/
> [2009-03-19 11:28:38,019]
> org.apache.activemq.transport.TransportServerThreadSupport INFO -
> Listening for connections at: tcp://berint.2e-systems.com:61616
> [2009-03-19 11:28:38,019] org.apache.activemq.broker.TransportConnector INFO
> - Connector tcp Started
> [2009-03-19 11:28:38,024] org.apache.activemq.broker.BrokerService INFO -
> ActiveMQ JMS Message Broker (localhost,
> ID:berint.2e-systems.com-47394-1237458517842-0:0) started
> [2009-03-19 11:28:38,048] org.apache.activemq.broker.jmx.ManagementContext
> INFO - JMX consoles can connect to
> service:jmx:rmi:///jndi/rmi://localhost:1099/jmxrmi
> [2009-03-19 11:28:38,464]
> org.apache.activemq.transport.vm.VMTransportFactory DEBUG - binding to
> broker: localhost
> [2009-03-19 11:28:38,465] org.apache.activemq.broker.TransportConnector INFO
> - Connector vm://localhost Started
> [2009-03-19 11:28:38,496] org.ap
>
> and camel starting:
>
> [2009-03-19 11:28:34,031] org.apache.camel.util.MainSupport INFO - Apache
> Camel 1.6.0 starting
> [2009-03-19 11:28:34,059]
> org.springframework.context.support.ClassPathXmlApplicationContext INFO -
> Refreshing
> org.springframework.context.support.ClassPathXmlApplicationContext@1df5a8f:
> display name
> [org.springframework.context.support.ClassPathXmlApplicationContext@1df5a8f];
> startup date [Thu Mar 19 11:28:34 CET 2009]; root of context hierarchy
> [2009-03-19 11:28:34,125]
> org.springframework.beans.factory.xml.XmlBeanDefinitionReader INFO -
> Loading XML bean definitions from class path resource [camel-server.xml]
> [2009-03-19 11:28:37,322]
> org.springframework.context.support.ClassPathXmlApplicationContext INFO -
> Bean factory for application context
> [org.springframework.context.support.ClassPathXmlA
> pplicationContext@1df5a8f]:
> org.springframework.beans.factory.support.DefaultListableBeanFactory@1fb3211
> [2009-03-19 11:28:37,475]
> org.springframework.context.support.ClassPathXmlApplicationContext INFO -
> Bean 'agent' is not eligible for getting processed by all BeanPostProcessors
> (for examp
> le: not eligible for auto-proxying)
> [2009-03-19 11:28:37,499] org.apache.camel.impl.DefaultCamelContext INFO -
> JMX enabled. Using InstrumentationLifecycleStrategy.
> [2009-03-19 11:28:37,513] org.apache.camel.spring.CamelContextFactoryBean
> DEBUG - JMXAgent enabled
> [2009-03-19 11:28:37,513] org.apache.camel.spring.CamelContextFactoryBean
> DEBUG - Found JAXB created routes: []
>
>
> Thanks
> Andrew
>
>
> Claus Ibsen-2 wrote:
>>
>> Hi
>>
>> That is fine. Then we know its AMQ loading the classes over and over
>> again.
>>
>> Can you post your AMQ configuration?
>>
>>
>> On Fri, Mar 20, 2009 at 11:30 AM, ee7arh <an...@2e-systems.com>
>> wrote:
>>>
>>> Hi,
>>>
>>> I can confirm from JHat that only a single instance of the bean classes
>>> are
>>> loaded:
>>>
>>> 1 instance of class com.ee.berbe.mobile.external.EventMarshalling
>>> 1 instance of class com.ee.berbe.mobile.external.EventRouter
>>> 1 instance of class
>>> com.ee.berbe.mobile.servicelogic.ServiceEventGenerator
>>> 1 instance of class com.ee.berbe.mobile.servicelogic.ServiceDataRouter
>>>
>>> If I look in JHat at the total number of classes loaded and order it
>>> ascending, if I search down for anything "camel" related I see:
>>>
>>> 121 instances of class
>>> org.apache.camel.impl.converter.DefaultTypeConverter$TypeMapping
>>>
>>> If I do the same for "activemq" I see many more instances:
>>>
>>> 4466 instances of class org.apache.activemq.command.SessionId
>>> 4456 instances of class org.apache.activemq.command.ConnectionId
>>> 4382 instances of class org.apache.activemq.command.ProducerId
>>> 3750 instances of class org.apache.activemq.util.ByteSequence
>>> 3748 instances of class org.apache.activemq.command.MessageId
>>> 3565 instances of class org.apache.activemq.command.ActiveMQQueue
>>>
>>> If I do the same for my internal object type I see:
>>>
>>> 3358 instances of class
>>> com.ee.berbe.mobile.servicelogic.xxxx.events.external.TriggerCkiInvites
>>>
>>>
>>> Does that answer your question satisfactorily or should I perform your
>>> tests
>>> to double check?
>>>
>>> Thanks
>>> Andrew
>>>
>>>
>>>
>>>
>>> Claus Ibsen-2 wrote:
>>>>
>>>> Hi
>>>>
>>>> Can you try using bean(Object, "methodname") in your route and pass in
>>>> the same object, just to be sure its not Camel loading your bean
>>>> constantly?
>>>> And you are sure your bean in Spring XML is defined as singleton scope?
>>>>
>>>> In 2.0 I have adding TRACE logging to ObjectHelper.loadClass that loads
>>>> a
>>>> class.
>>>> Then we will be able to TRACE if Camel is loading the same class over
>>>> and over again.
>>>>
>>>>
>>>>
>>>> Another try is to use .process() instead of bean and put the code in
>>>> there so you are sure the class is not loaded.
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/References-to-objects-processed-in-routes-held%2C-eventually-run-out-of-memory-tp22580791p22617746.html
>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>>
>> --
>> Claus Ibsen
>> Apache Camel Committer
>>
>> Open Source Integration: http://fusesource.com
>> Blog: http://davsclaus.blogspot.com/
>>
>>
>
> --
> View this message in context: http://www.nabble.com/References-to-objects-processed-in-routes-held%2C-eventually-run-out-of-memory-tp22580791p22617943.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>
--
Claus Ibsen
Apache Camel Committer
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Re: References to objects processed in routes held, eventually run
out of memory
Posted by ee7arh <an...@2e-systems.com>.
Hi,
Here it comes:
<!-- A local ativeMQ broker -->
<broker:broker useJmx="true" persistent="false" brokerName="localhost">
<broker:transportConnectors>
<broker:transportConnector name="tcp" uri="tcp://localhost:61616"/>
</broker:transportConnectors>
</broker:broker>
<!-- Connection factory for ActiveMQ endpoints -->
<bean id="connectionFactoryActiveMQ"
class="org.apache.activemq.ActiveMQConnectionFactory">
<property name="brokerURL" value="vm://localhost"/>
</bean>
<!-- configure the Camel JMS consumer to use the ActiveMQ broker declared
above -->
<bean id="jms" class="org.apache.camel.component.jms.JmsComponent">
<property name="connectionFactory" ref="connectionFactoryActiveMQ"/>
</bean>
and my camel context for completeness:
<camel:camelContext id="camel"
xmlns="http://activemq.apache.org/camel/schema/spring">
<camel:package>com.ee.berbe.mobile</camel:package>
<!-- enable JMX connector so we can connect to the server and
browse mbeans -->
<camel:jmxAgent id="agent" createConnector="true"
registryPort="1098"/>
</camel:camelContext>
here is ActiveMQ starting:
[2009-03-19 11:28:37,785] org.apache.activemq.broker.BrokerService INFO -
ActiveMQ 5.2.0 JMS Message Broker (localhost) is starting
[2009-03-19 11:28:37,785] org.apache.activemq.broker.BrokerService INFO -
For help or more information please see: http://activemq.apache.org/
[2009-03-19 11:28:38,019]
org.apache.activemq.transport.TransportServerThreadSupport INFO -
Listening for connections at: tcp://berint.2e-systems.com:61616
[2009-03-19 11:28:38,019] org.apache.activemq.broker.TransportConnector INFO
- Connector tcp Started
[2009-03-19 11:28:38,024] org.apache.activemq.broker.BrokerService INFO -
ActiveMQ JMS Message Broker (localhost,
ID:berint.2e-systems.com-47394-1237458517842-0:0) started
[2009-03-19 11:28:38,048] org.apache.activemq.broker.jmx.ManagementContext
INFO - JMX consoles can connect to
service:jmx:rmi:///jndi/rmi://localhost:1099/jmxrmi
[2009-03-19 11:28:38,464]
org.apache.activemq.transport.vm.VMTransportFactory DEBUG - binding to
broker: localhost
[2009-03-19 11:28:38,465] org.apache.activemq.broker.TransportConnector INFO
- Connector vm://localhost Started
[2009-03-19 11:28:38,496] org.ap
and camel starting:
[2009-03-19 11:28:34,031] org.apache.camel.util.MainSupport INFO - Apache
Camel 1.6.0 starting
[2009-03-19 11:28:34,059]
org.springframework.context.support.ClassPathXmlApplicationContext INFO -
Refreshing
org.springframework.context.support.ClassPathXmlApplicationContext@1df5a8f:
display name
[org.springframework.context.support.ClassPathXmlApplicationContext@1df5a8f];
startup date [Thu Mar 19 11:28:34 CET 2009]; root of context hierarchy
[2009-03-19 11:28:34,125]
org.springframework.beans.factory.xml.XmlBeanDefinitionReader INFO -
Loading XML bean definitions from class path resource [camel-server.xml]
[2009-03-19 11:28:37,322]
org.springframework.context.support.ClassPathXmlApplicationContext INFO -
Bean factory for application context
[org.springframework.context.support.ClassPathXmlA
pplicationContext@1df5a8f]:
org.springframework.beans.factory.support.DefaultListableBeanFactory@1fb3211
[2009-03-19 11:28:37,475]
org.springframework.context.support.ClassPathXmlApplicationContext INFO -
Bean 'agent' is not eligible for getting processed by all BeanPostProcessors
(for examp
le: not eligible for auto-proxying)
[2009-03-19 11:28:37,499] org.apache.camel.impl.DefaultCamelContext INFO -
JMX enabled. Using InstrumentationLifecycleStrategy.
[2009-03-19 11:28:37,513] org.apache.camel.spring.CamelContextFactoryBean
DEBUG - JMXAgent enabled
[2009-03-19 11:28:37,513] org.apache.camel.spring.CamelContextFactoryBean
DEBUG - Found JAXB created routes: []
Thanks
Andrew
Claus Ibsen-2 wrote:
>
> Hi
>
> That is fine. Then we know its AMQ loading the classes over and over
> again.
>
> Can you post your AMQ configuration?
>
>
> On Fri, Mar 20, 2009 at 11:30 AM, ee7arh <an...@2e-systems.com>
> wrote:
>>
>> Hi,
>>
>> I can confirm from JHat that only a single instance of the bean classes
>> are
>> loaded:
>>
>> 1 instance of class com.ee.berbe.mobile.external.EventMarshalling
>> 1 instance of class com.ee.berbe.mobile.external.EventRouter
>> 1 instance of class
>> com.ee.berbe.mobile.servicelogic.ServiceEventGenerator
>> 1 instance of class com.ee.berbe.mobile.servicelogic.ServiceDataRouter
>>
>> If I look in JHat at the total number of classes loaded and order it
>> ascending, if I search down for anything "camel" related I see:
>>
>> 121 instances of class
>> org.apache.camel.impl.converter.DefaultTypeConverter$TypeMapping
>>
>> If I do the same for "activemq" I see many more instances:
>>
>> 4466 instances of class org.apache.activemq.command.SessionId
>> 4456 instances of class org.apache.activemq.command.ConnectionId
>> 4382 instances of class org.apache.activemq.command.ProducerId
>> 3750 instances of class org.apache.activemq.util.ByteSequence
>> 3748 instances of class org.apache.activemq.command.MessageId
>> 3565 instances of class org.apache.activemq.command.ActiveMQQueue
>>
>> If I do the same for my internal object type I see:
>>
>> 3358 instances of class
>> com.ee.berbe.mobile.servicelogic.xxxx.events.external.TriggerCkiInvites
>>
>>
>> Does that answer your question satisfactorily or should I perform your
>> tests
>> to double check?
>>
>> Thanks
>> Andrew
>>
>>
>>
>>
>> Claus Ibsen-2 wrote:
>>>
>>> Hi
>>>
>>> Can you try using bean(Object, "methodname") in your route and pass in
>>> the same object, just to be sure its not Camel loading your bean
>>> constantly?
>>> And you are sure your bean in Spring XML is defined as singleton scope?
>>>
>>> In 2.0 I have adding TRACE logging to ObjectHelper.loadClass that loads
>>> a
>>> class.
>>> Then we will be able to TRACE if Camel is loading the same class over
>>> and over again.
>>>
>>>
>>>
>>> Another try is to use .process() instead of bean and put the code in
>>> there so you are sure the class is not loaded.
>>>
>>>
>>
>>
>>
>> --
>> View this message in context:
>> http://www.nabble.com/References-to-objects-processed-in-routes-held%2C-eventually-run-out-of-memory-tp22580791p22617746.html
>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>
>>
>
>
>
> --
> Claus Ibsen
> Apache Camel Committer
>
> Open Source Integration: http://fusesource.com
> Blog: http://davsclaus.blogspot.com/
>
>
--
View this message in context: http://www.nabble.com/References-to-objects-processed-in-routes-held%2C-eventually-run-out-of-memory-tp22580791p22617943.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: References to objects processed in routes held, eventually run
out of memory
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
That is fine. Then we know its AMQ loading the classes over and over again.
Can you post your AMQ configuration?
On Fri, Mar 20, 2009 at 11:30 AM, ee7arh <an...@2e-systems.com> wrote:
>
> Hi,
>
> I can confirm from JHat that only a single instance of the bean classes are
> loaded:
>
> 1 instance of class com.ee.berbe.mobile.external.EventMarshalling
> 1 instance of class com.ee.berbe.mobile.external.EventRouter
> 1 instance of class com.ee.berbe.mobile.servicelogic.ServiceEventGenerator
> 1 instance of class com.ee.berbe.mobile.servicelogic.ServiceDataRouter
>
> If I look in JHat at the total number of classes loaded and order it
> ascending, if I search down for anything "camel" related I see:
>
> 121 instances of class
> org.apache.camel.impl.converter.DefaultTypeConverter$TypeMapping
>
> If I do the same for "activemq" I see many more instances:
>
> 4466 instances of class org.apache.activemq.command.SessionId
> 4456 instances of class org.apache.activemq.command.ConnectionId
> 4382 instances of class org.apache.activemq.command.ProducerId
> 3750 instances of class org.apache.activemq.util.ByteSequence
> 3748 instances of class org.apache.activemq.command.MessageId
> 3565 instances of class org.apache.activemq.command.ActiveMQQueue
>
> If I do the same for my internal object type I see:
>
> 3358 instances of class
> com.ee.berbe.mobile.servicelogic.xxxx.events.external.TriggerCkiInvites
>
>
> Does that answer your question satisfactorily or should I perform your tests
> to double check?
>
> Thanks
> Andrew
>
>
>
>
> Claus Ibsen-2 wrote:
>>
>> Hi
>>
>> Can you try using bean(Object, "methodname") in your route and pass in
>> the same object, just to be sure its not Camel loading your bean
>> constantly?
>> And you are sure your bean in Spring XML is defined as singleton scope?
>>
>> In 2.0 I have adding TRACE logging to ObjectHelper.loadClass that loads a
>> class.
>> Then we will be able to TRACE if Camel is loading the same class over
>> and over again.
>>
>>
>>
>> Another try is to use .process() instead of bean and put the code in
>> there so you are sure the class is not loaded.
>>
>>
>
>
>
> --
> View this message in context: http://www.nabble.com/References-to-objects-processed-in-routes-held%2C-eventually-run-out-of-memory-tp22580791p22617746.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>
--
Claus Ibsen
Apache Camel Committer
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Re: References to objects processed in routes held, eventually run
out of memory
Posted by ee7arh <an...@2e-systems.com>.
Hi,
I can confirm from JHat that only a single instance of the bean classes are
loaded:
1 instance of class com.ee.berbe.mobile.external.EventMarshalling
1 instance of class com.ee.berbe.mobile.external.EventRouter
1 instance of class com.ee.berbe.mobile.servicelogic.ServiceEventGenerator
1 instance of class com.ee.berbe.mobile.servicelogic.ServiceDataRouter
If I look in JHat at the total number of classes loaded and order it
ascending, if I search down for anything "camel" related I see:
121 instances of class
org.apache.camel.impl.converter.DefaultTypeConverter$TypeMapping
If I do the same for "activemq" I see many more instances:
4466 instances of class org.apache.activemq.command.SessionId
4456 instances of class org.apache.activemq.command.ConnectionId
4382 instances of class org.apache.activemq.command.ProducerId
3750 instances of class org.apache.activemq.util.ByteSequence
3748 instances of class org.apache.activemq.command.MessageId
3565 instances of class org.apache.activemq.command.ActiveMQQueue
If I do the same for my internal object type I see:
3358 instances of class
com.ee.berbe.mobile.servicelogic.xxxx.events.external.TriggerCkiInvites
Does that answer your question satisfactorily or should I perform your tests
to double check?
Thanks
Andrew
Claus Ibsen-2 wrote:
>
> Hi
>
> Can you try using bean(Object, "methodname") in your route and pass in
> the same object, just to be sure its not Camel loading your bean
> constantly?
> And you are sure your bean in Spring XML is defined as singleton scope?
>
> In 2.0 I have adding TRACE logging to ObjectHelper.loadClass that loads a
> class.
> Then we will be able to TRACE if Camel is loading the same class over
> and over again.
>
>
>
> Another try is to use .process() instead of bean and put the code in
> there so you are sure the class is not loaded.
>
>
--
View this message in context: http://www.nabble.com/References-to-objects-processed-in-routes-held%2C-eventually-run-out-of-memory-tp22580791p22617746.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: References to objects processed in routes held, eventually run
out of memory
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
Can you try using bean(Object, "methodname") in your route and pass in
the same object, just to be sure its not Camel loading your bean
constantly?
And you are sure your bean in Spring XML is defined as singleton scope?
In 2.0 I have adding TRACE logging to ObjectHelper.loadClass that loads a class.
Then we will be able to TRACE if Camel is loading the same class over
and over again.
Another try is to use .process() instead of bean and put the code in
there so you are sure the class is not loaded.
On Thu, Mar 19, 2009 at 2:24 PM, ee7arh <an...@2e-systems.com> wrote:
>
> Hi,
>
> Thanks for response.
>
> I'm using Camel 1.6.0 and ActiveMQ 5.2.0. I also had the same results with
> camel 1.5 and activeMQ 5.1
>
> I have attached a graph from Jconsole showing what happens as the
> application starts, runs for a about 1.5 hours then crashes with out of
> memory. http://www.nabble.com/file/p22600122/jconsole-graphs.jpg
>
> Notes:
>
> - I set the maximum memory to 64MB using -Xmx64M -server. normally I would
> use more but then the problem takes longer to reproduce
> - Notice the number of loaded classes never reduces. I also have another
> application which shows the same pattern.
> - As the application starts to run out of memory, the CPU usage goes
> extremely high
>
> Here is my route:
>
> from("jms:queue:mobileExtTriggerInvites")
>
> .errorHandler(deadLetterChannel("seda:errors").maximumRedeliveries(0))
>
> // Exception Handling
> .onException(JAXBException.class)
> .to("jms:queue:unmarshallableRequest").end()
> .onException(UnexpectedEventException.class)
> .to("jms:queue:unexpectedEvent").end()
>
> // Log Incoming XML
> .to("log:incomingEventXmlLogger?level=INFO")
>
> // Attempt to Unmarshall object expected on this queue
> .to("bean:eventMarshaller?methodName=unmarshallTriggerInvites")
>
> // Generate Service Events from this External Event
>
> .to("bean:serviceEventGenerator?methodName=procecssTriggerInvites")
>
> // Split generated Events out of List into individual events
> .splitter(body())
>
> // Destination queue for further processing
> .to("jms:queue:serviceEventQueue");
>
> You can see that I have 2 bean processors, I have also attached the code for
> those.
>
> http://www.nabble.com/file/p22600122/EventMarhaller.txt EventMarhaller.txt
> http://www.nabble.com/file/p22600122/ServiceEventGenerator.txt
> ServiceEventGenerator.txt
>
> Perhaps it's useful, perhaps not but I read a blog article which sounded
> very similar to the problem I am having, link below. Basically it looks like
> Camel or ActiveMQ still has a reference to every single object I create and
> send through the routing - this would surely explain why the number of
> loaded classes gets bigger and bigger the more the application runs.
>
> http://dertompson.com/2007/08/01/fighting-the-outofmemoryerror/
> http://dertompson.com/2007/08/01/fighting-the-outofmemoryerror
>
> using JConsole, I also periodically purged the end queue and this made no
> difference.
>
> If it helps, I am able to provide a heapdump to anyone for analysis with
> jHat. As mentioned in my 1st post, all instances of classes created during
> the application lifecycle continue to live on in memory. It appears that my
> objects are referenced by ActiveMQMessage
>
> For example, I will trace 1 of those classes created in code attached:
>
> Class:
> class
> com.ee.berbe.mobile.servicelogic.flightops.events.service.TriggerMMSInvitations
>
> References to this object:
> org.apache.activemq.command.ActiveMQObjectMessage@0xafab8fd8 (174 bytes) :
> field object
>
> --> -->
>
> Class:
> class org.apache.activemq.command.ActiveMQObjectMessage
>
> References to this object:
> java.util.LinkedHashMap$Entry@0xafab8970 (32 bytes) : field value
> java.util.LinkedHashMap$Entry@0xafab8810 (32 bytes) : field value
>
>
> So it seems that ActiveMQ is still storing all my objets on a LinkedHashMap
> but I've no idea why.
>
> Thanks again for any help
>
> Andrew
>
>
> willem.jiang wrote:
>>
>> Hi
>> Which version of Camel are you using ?
>> Can you show us you routing rules and your processor codes?
>>
>> They will help us to dig the issue.
>>
>> Willem
>>
>>
> --
> View this message in context: http://www.nabble.com/References-to-objects-processed-in-routes-held%2C-eventually-run-out-of-memory-tp22580791p22600122.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>
--
Claus Ibsen
Apache Camel Committer
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Re: References to objects processed in routes held, eventually run
out of memory
Posted by ee7arh <an...@2e-systems.com>.
Hi again,
A bit more information which might help. I have been running the application
for the last 4 hours with 256MB memory and so far it looks like the
outOfMemory problem will probably not occur anytime soon because the heap
usage drops dramatically after the initial high load processing.
In addition, I have been running messages through further routes to perform
more processing. In my last example, i ran the applciation only through a
simple route but using less memory
Here is the graph:
http://www.nabble.com/file/p22601000/jconsole-graphs-2.jpg
When the application was first started it was under high load so you can see
it took almsot 200MB but after the first 30 mns, fewer messages were
processed and so the application looks to gradually recover the memory.
however, having said all that, notice once again the classes loaded into
memory, it does not follow the same pattern. Number of loaded classes keeps
on increasing so I guess although it may now take a long time, the
application will eventually run out of memory.
Perhaps then there are 2 reasons which can cause an out of memory:
1) When I simply do not have enough memory to facilitate high-load procesing
2) When the nunber of classes in memory continues to increase until no more
memory is left...
Andrew
ee7arh wrote:
>
> Hi,
>
> Thanks but we are not using camel-jaxb, everything is being done manually
> using the JaxB classes directly (although now I see camel has this built
> in I will change.... well for R2.0 anyway!)
>
> As way of double checking, I found only 7 instances of my unmarshalled
> objects in the memory dump, not 1600 like I see for the aforementioned
> object in previous post.
>
> Andrew
>
>
> Claus Ibsen-2 wrote:
>>
>> Hi
>>
>> Just a very quick reply without digging into it yet. There was/is a
>> leak in camel-jaxb.
>> I cant remember if we got it fixed in 1.6.0 or its in 1.6.1. At least
>> its fixed in 2.0m1.
>>
>> Could you either:
>> - try without any JAXB
>> - try with 1.6.1-SNAPSHOT or 2.0m1/2.0-SNAPSHOT
>>
>>
>>
>> On Thu, Mar 19, 2009 at 2:24 PM, ee7arh <an...@2e-systems.com>
>> wrote:
>>>
>>> Hi,
>>>
>>> Thanks for response.
>>>
>>> I'm using Camel 1.6.0 and ActiveMQ 5.2.0. I also had the same results
>>> with
>>> camel 1.5 and activeMQ 5.1
>>>
>>> I have attached a graph from Jconsole showing what happens as the
>>> application starts, runs for a about 1.5 hours then crashes with out of
>>> memory. http://www.nabble.com/file/p22600122/jconsole-graphs.jpg
>>>
>>> Notes:
>>>
>>> - I set the maximum memory to 64MB using -Xmx64M -server. normally I
>>> would
>>> use more but then the problem takes longer to reproduce
>>> - Notice the number of loaded classes never reduces. I also have another
>>> application which shows the same pattern.
>>> - As the application starts to run out of memory, the CPU usage goes
>>> extremely high
>>>
>>> Here is my route:
>>>
>>> from("jms:queue:mobileExtTriggerInvites")
>>>
>>> .errorHandler(deadLetterChannel("seda:errors").maximumRedeliveries(0))
>>>
>>> // Exception Handling
>>> .onException(JAXBException.class)
>>> .to("jms:queue:unmarshallableRequest").end()
>>> .onException(UnexpectedEventException.class)
>>> .to("jms:queue:unexpectedEvent").end()
>>>
>>> // Log Incoming XML
>>> .to("log:incomingEventXmlLogger?level=INFO")
>>>
>>> // Attempt to Unmarshall object expected on this queue
>>>
>>> .to("bean:eventMarshaller?methodName=unmarshallTriggerInvites")
>>>
>>> // Generate Service Events from this External Event
>>>
>>> .to("bean:serviceEventGenerator?methodName=procecssTriggerInvites")
>>>
>>> // Split generated Events out of List into individual events
>>> .splitter(body())
>>>
>>> // Destination queue for further processing
>>> .to("jms:queue:serviceEventQueue");
>>>
>>> You can see that I have 2 bean processors, I have also attached the code
>>> for
>>> those.
>>>
>>> http://www.nabble.com/file/p22600122/EventMarhaller.txt
>>> EventMarhaller.txt
>>> http://www.nabble.com/file/p22600122/ServiceEventGenerator.txt
>>> ServiceEventGenerator.txt
>>>
>>> Perhaps it's useful, perhaps not but I read a blog article which sounded
>>> very similar to the problem I am having, link below. Basically it looks
>>> like
>>> Camel or ActiveMQ still has a reference to every single object I create
>>> and
>>> send through the routing - this would surely explain why the number of
>>> loaded classes gets bigger and bigger the more the application runs.
>>>
>>> http://dertompson.com/2007/08/01/fighting-the-outofmemoryerror/
>>> http://dertompson.com/2007/08/01/fighting-the-outofmemoryerror
>>>
>>> using JConsole, I also periodically purged the end queue and this made
>>> no
>>> difference.
>>>
>>> If it helps, I am able to provide a heapdump to anyone for analysis with
>>> jHat. As mentioned in my 1st post, all instances of classes created
>>> during
>>> the application lifecycle continue to live on in memory. It appears that
>>> my
>>> objects are referenced by ActiveMQMessage
>>>
>>> For example, I will trace 1 of those classes created in code attached:
>>>
>>> Class:
>>> class
>>> com.ee.berbe.mobile.servicelogic.flightops.events.service.TriggerMMSInvitations
>>>
>>> References to this object:
>>> org.apache.activemq.command.ActiveMQObjectMessage@0xafab8fd8 (174 bytes)
>>> :
>>> field object
>>>
>>> --> -->
>>>
>>> Class:
>>> class org.apache.activemq.command.ActiveMQObjectMessage
>>>
>>> References to this object:
>>> java.util.LinkedHashMap$Entry@0xafab8970 (32 bytes) : field value
>>> java.util.LinkedHashMap$Entry@0xafab8810 (32 bytes) : field value
>>>
>>>
>>> So it seems that ActiveMQ is still storing all my objets on a
>>> LinkedHashMap
>>> but I've no idea why.
>>>
>>> Thanks again for any help
>>>
>>> Andrew
>>>
>>>
>>> willem.jiang wrote:
>>>>
>>>> Hi
>>>> Which version of Camel are you using ?
>>>> Can you show us you routing rules and your processor codes?
>>>>
>>>> They will help us to dig the issue.
>>>>
>>>> Willem
>>>>
>>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/References-to-objects-processed-in-routes-held%2C-eventually-run-out-of-memory-tp22580791p22600122.html
>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>>
>> --
>> Claus Ibsen
>> Apache Camel Committer
>>
>> Open Source Integration: http://fusesource.com
>> Blog: http://davsclaus.blogspot.com/
>>
>>
>
>
--
View this message in context: http://www.nabble.com/References-to-objects-processed-in-routes-held%2C-eventually-run-out-of-memory-tp22580791p22601000.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: References to objects processed in routes held, eventually run
out of memory
Posted by ee7arh <an...@2e-systems.com>.
Hi,
Thanks but we are not using camel-jaxb, everything is being done manually
using the JaxB classes directly (although now I see camel has this built in
I will change.... well for R2.0 anyway!)
As way of double checking, I found only 7 instances of my unmarshalled
objects in the memory dump, not 1600 like I see for the aforementioned
object in previous post.
Andrew
Claus Ibsen-2 wrote:
>
> Hi
>
> Just a very quick reply without digging into it yet. There was/is a
> leak in camel-jaxb.
> I cant remember if we got it fixed in 1.6.0 or its in 1.6.1. At least
> its fixed in 2.0m1.
>
> Could you either:
> - try without any JAXB
> - try with 1.6.1-SNAPSHOT or 2.0m1/2.0-SNAPSHOT
>
>
>
> On Thu, Mar 19, 2009 at 2:24 PM, ee7arh <an...@2e-systems.com>
> wrote:
>>
>> Hi,
>>
>> Thanks for response.
>>
>> I'm using Camel 1.6.0 and ActiveMQ 5.2.0. I also had the same results
>> with
>> camel 1.5 and activeMQ 5.1
>>
>> I have attached a graph from Jconsole showing what happens as the
>> application starts, runs for a about 1.5 hours then crashes with out of
>> memory. http://www.nabble.com/file/p22600122/jconsole-graphs.jpg
>>
>> Notes:
>>
>> - I set the maximum memory to 64MB using -Xmx64M -server. normally I
>> would
>> use more but then the problem takes longer to reproduce
>> - Notice the number of loaded classes never reduces. I also have another
>> application which shows the same pattern.
>> - As the application starts to run out of memory, the CPU usage goes
>> extremely high
>>
>> Here is my route:
>>
>> from("jms:queue:mobileExtTriggerInvites")
>>
>> .errorHandler(deadLetterChannel("seda:errors").maximumRedeliveries(0))
>>
>> // Exception Handling
>> .onException(JAXBException.class)
>> .to("jms:queue:unmarshallableRequest").end()
>> .onException(UnexpectedEventException.class)
>> .to("jms:queue:unexpectedEvent").end()
>>
>> // Log Incoming XML
>> .to("log:incomingEventXmlLogger?level=INFO")
>>
>> // Attempt to Unmarshall object expected on this queue
>>
>> .to("bean:eventMarshaller?methodName=unmarshallTriggerInvites")
>>
>> // Generate Service Events from this External Event
>>
>> .to("bean:serviceEventGenerator?methodName=procecssTriggerInvites")
>>
>> // Split generated Events out of List into individual events
>> .splitter(body())
>>
>> // Destination queue for further processing
>> .to("jms:queue:serviceEventQueue");
>>
>> You can see that I have 2 bean processors, I have also attached the code
>> for
>> those.
>>
>> http://www.nabble.com/file/p22600122/EventMarhaller.txt
>> EventMarhaller.txt
>> http://www.nabble.com/file/p22600122/ServiceEventGenerator.txt
>> ServiceEventGenerator.txt
>>
>> Perhaps it's useful, perhaps not but I read a blog article which sounded
>> very similar to the problem I am having, link below. Basically it looks
>> like
>> Camel or ActiveMQ still has a reference to every single object I create
>> and
>> send through the routing - this would surely explain why the number of
>> loaded classes gets bigger and bigger the more the application runs.
>>
>> http://dertompson.com/2007/08/01/fighting-the-outofmemoryerror/
>> http://dertompson.com/2007/08/01/fighting-the-outofmemoryerror
>>
>> using JConsole, I also periodically purged the end queue and this made no
>> difference.
>>
>> If it helps, I am able to provide a heapdump to anyone for analysis with
>> jHat. As mentioned in my 1st post, all instances of classes created
>> during
>> the application lifecycle continue to live on in memory. It appears that
>> my
>> objects are referenced by ActiveMQMessage
>>
>> For example, I will trace 1 of those classes created in code attached:
>>
>> Class:
>> class
>> com.ee.berbe.mobile.servicelogic.flightops.events.service.TriggerMMSInvitations
>>
>> References to this object:
>> org.apache.activemq.command.ActiveMQObjectMessage@0xafab8fd8 (174 bytes)
>> :
>> field object
>>
>> --> -->
>>
>> Class:
>> class org.apache.activemq.command.ActiveMQObjectMessage
>>
>> References to this object:
>> java.util.LinkedHashMap$Entry@0xafab8970 (32 bytes) : field value
>> java.util.LinkedHashMap$Entry@0xafab8810 (32 bytes) : field value
>>
>>
>> So it seems that ActiveMQ is still storing all my objets on a
>> LinkedHashMap
>> but I've no idea why.
>>
>> Thanks again for any help
>>
>> Andrew
>>
>>
>> willem.jiang wrote:
>>>
>>> Hi
>>> Which version of Camel are you using ?
>>> Can you show us you routing rules and your processor codes?
>>>
>>> They will help us to dig the issue.
>>>
>>> Willem
>>>
>>>
>> --
>> View this message in context:
>> http://www.nabble.com/References-to-objects-processed-in-routes-held%2C-eventually-run-out-of-memory-tp22580791p22600122.html
>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>
>>
>
>
>
> --
> Claus Ibsen
> Apache Camel Committer
>
> Open Source Integration: http://fusesource.com
> Blog: http://davsclaus.blogspot.com/
>
>
--
View this message in context: http://www.nabble.com/References-to-objects-processed-in-routes-held%2C-eventually-run-out-of-memory-tp22580791p22600424.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: References to objects processed in routes held, eventually run
out of memory
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
Just a very quick reply without digging into it yet. There was/is a
leak in camel-jaxb.
I cant remember if we got it fixed in 1.6.0 or its in 1.6.1. At least
its fixed in 2.0m1.
Could you either:
- try without any JAXB
- try with 1.6.1-SNAPSHOT or 2.0m1/2.0-SNAPSHOT
On Thu, Mar 19, 2009 at 2:24 PM, ee7arh <an...@2e-systems.com> wrote:
>
> Hi,
>
> Thanks for response.
>
> I'm using Camel 1.6.0 and ActiveMQ 5.2.0. I also had the same results with
> camel 1.5 and activeMQ 5.1
>
> I have attached a graph from Jconsole showing what happens as the
> application starts, runs for a about 1.5 hours then crashes with out of
> memory. http://www.nabble.com/file/p22600122/jconsole-graphs.jpg
>
> Notes:
>
> - I set the maximum memory to 64MB using -Xmx64M -server. normally I would
> use more but then the problem takes longer to reproduce
> - Notice the number of loaded classes never reduces. I also have another
> application which shows the same pattern.
> - As the application starts to run out of memory, the CPU usage goes
> extremely high
>
> Here is my route:
>
> from("jms:queue:mobileExtTriggerInvites")
>
> .errorHandler(deadLetterChannel("seda:errors").maximumRedeliveries(0))
>
> // Exception Handling
> .onException(JAXBException.class)
> .to("jms:queue:unmarshallableRequest").end()
> .onException(UnexpectedEventException.class)
> .to("jms:queue:unexpectedEvent").end()
>
> // Log Incoming XML
> .to("log:incomingEventXmlLogger?level=INFO")
>
> // Attempt to Unmarshall object expected on this queue
> .to("bean:eventMarshaller?methodName=unmarshallTriggerInvites")
>
> // Generate Service Events from this External Event
>
> .to("bean:serviceEventGenerator?methodName=procecssTriggerInvites")
>
> // Split generated Events out of List into individual events
> .splitter(body())
>
> // Destination queue for further processing
> .to("jms:queue:serviceEventQueue");
>
> You can see that I have 2 bean processors, I have also attached the code for
> those.
>
> http://www.nabble.com/file/p22600122/EventMarhaller.txt EventMarhaller.txt
> http://www.nabble.com/file/p22600122/ServiceEventGenerator.txt
> ServiceEventGenerator.txt
>
> Perhaps it's useful, perhaps not but I read a blog article which sounded
> very similar to the problem I am having, link below. Basically it looks like
> Camel or ActiveMQ still has a reference to every single object I create and
> send through the routing - this would surely explain why the number of
> loaded classes gets bigger and bigger the more the application runs.
>
> http://dertompson.com/2007/08/01/fighting-the-outofmemoryerror/
> http://dertompson.com/2007/08/01/fighting-the-outofmemoryerror
>
> using JConsole, I also periodically purged the end queue and this made no
> difference.
>
> If it helps, I am able to provide a heapdump to anyone for analysis with
> jHat. As mentioned in my 1st post, all instances of classes created during
> the application lifecycle continue to live on in memory. It appears that my
> objects are referenced by ActiveMQMessage
>
> For example, I will trace 1 of those classes created in code attached:
>
> Class:
> class
> com.ee.berbe.mobile.servicelogic.flightops.events.service.TriggerMMSInvitations
>
> References to this object:
> org.apache.activemq.command.ActiveMQObjectMessage@0xafab8fd8 (174 bytes) :
> field object
>
> --> -->
>
> Class:
> class org.apache.activemq.command.ActiveMQObjectMessage
>
> References to this object:
> java.util.LinkedHashMap$Entry@0xafab8970 (32 bytes) : field value
> java.util.LinkedHashMap$Entry@0xafab8810 (32 bytes) : field value
>
>
> So it seems that ActiveMQ is still storing all my objets on a LinkedHashMap
> but I've no idea why.
>
> Thanks again for any help
>
> Andrew
>
>
> willem.jiang wrote:
>>
>> Hi
>> Which version of Camel are you using ?
>> Can you show us you routing rules and your processor codes?
>>
>> They will help us to dig the issue.
>>
>> Willem
>>
>>
> --
> View this message in context: http://www.nabble.com/References-to-objects-processed-in-routes-held%2C-eventually-run-out-of-memory-tp22580791p22600122.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>
--
Claus Ibsen
Apache Camel Committer
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Re: References to objects processed in routes held, eventually run
out of memory
Posted by ee7arh <an...@2e-systems.com>.
Hi,
Thanks for response.
I'm using Camel 1.6.0 and ActiveMQ 5.2.0. I also had the same results with
camel 1.5 and activeMQ 5.1
I have attached a graph from Jconsole showing what happens as the
application starts, runs for a about 1.5 hours then crashes with out of
memory. http://www.nabble.com/file/p22600122/jconsole-graphs.jpg
Notes:
- I set the maximum memory to 64MB using -Xmx64M -server. normally I would
use more but then the problem takes longer to reproduce
- Notice the number of loaded classes never reduces. I also have another
application which shows the same pattern.
- As the application starts to run out of memory, the CPU usage goes
extremely high
Here is my route:
from("jms:queue:mobileExtTriggerInvites")
.errorHandler(deadLetterChannel("seda:errors").maximumRedeliveries(0))
// Exception Handling
.onException(JAXBException.class)
.to("jms:queue:unmarshallableRequest").end()
.onException(UnexpectedEventException.class)
.to("jms:queue:unexpectedEvent").end()
// Log Incoming XML
.to("log:incomingEventXmlLogger?level=INFO")
// Attempt to Unmarshall object expected on this queue
.to("bean:eventMarshaller?methodName=unmarshallTriggerInvites")
// Generate Service Events from this External Event
.to("bean:serviceEventGenerator?methodName=procecssTriggerInvites")
// Split generated Events out of List into individual events
.splitter(body())
// Destination queue for further processing
.to("jms:queue:serviceEventQueue");
You can see that I have 2 bean processors, I have also attached the code for
those.
http://www.nabble.com/file/p22600122/EventMarhaller.txt EventMarhaller.txt
http://www.nabble.com/file/p22600122/ServiceEventGenerator.txt
ServiceEventGenerator.txt
Perhaps it's useful, perhaps not but I read a blog article which sounded
very similar to the problem I am having, link below. Basically it looks like
Camel or ActiveMQ still has a reference to every single object I create and
send through the routing - this would surely explain why the number of
loaded classes gets bigger and bigger the more the application runs.
http://dertompson.com/2007/08/01/fighting-the-outofmemoryerror/
http://dertompson.com/2007/08/01/fighting-the-outofmemoryerror
using JConsole, I also periodically purged the end queue and this made no
difference.
If it helps, I am able to provide a heapdump to anyone for analysis with
jHat. As mentioned in my 1st post, all instances of classes created during
the application lifecycle continue to live on in memory. It appears that my
objects are referenced by ActiveMQMessage
For example, I will trace 1 of those classes created in code attached:
Class:
class
com.ee.berbe.mobile.servicelogic.flightops.events.service.TriggerMMSInvitations
References to this object:
org.apache.activemq.command.ActiveMQObjectMessage@0xafab8fd8 (174 bytes) :
field object
--> -->
Class:
class org.apache.activemq.command.ActiveMQObjectMessage
References to this object:
java.util.LinkedHashMap$Entry@0xafab8970 (32 bytes) : field value
java.util.LinkedHashMap$Entry@0xafab8810 (32 bytes) : field value
So it seems that ActiveMQ is still storing all my objets on a LinkedHashMap
but I've no idea why.
Thanks again for any help
Andrew
willem.jiang wrote:
>
> Hi
> Which version of Camel are you using ?
> Can you show us you routing rules and your processor codes?
>
> They will help us to dig the issue.
>
> Willem
>
>
--
View this message in context: http://www.nabble.com/References-to-objects-processed-in-routes-held%2C-eventually-run-out-of-memory-tp22580791p22600122.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: References to objects processed in routes held, eventually run
out of memory
Posted by Willem Jiang <wi...@gmail.com>.
Hi
Which version of Camel are you using ?
Can you show us you routing rules and your processor codes?
They will help us to dig the issue.
Willem
ee7arh wrote:
> Hi,
>
> Just a bit more info....
>
> I am still testing and waiting to see if the out-of-memory occurs again even
> though i am setting now the jvm option: -Xmx32M -server
>
> I am monitoring my application through jconsole and I notice the the number
> of "loaded classes" continues to increase and increase. I monitored another
> application we have which is stable and notice the same pattern although the
> total java heap size stays more or less the same.
>
> Should i be worried that the loaded classes increases? In any case, the java
> heap keeps steadily increasing, although it does drop when I invoke a manual
> garbage collect.
>
> Any ideas would be appreciated.
>
> thanks
> Andrew
Re: References to objects processed in routes held, eventually run
out of memory
Posted by ee7arh <an...@2e-systems.com>.
Hi,
Just a bit more info....
I am still testing and waiting to see if the out-of-memory occurs again even
though i am setting now the jvm option: -Xmx32M -server
I am monitoring my application through jconsole and I notice the the number
of "loaded classes" continues to increase and increase. I monitored another
application we have which is stable and notice the same pattern although the
total java heap size stays more or less the same.
Should i be worried that the loaded classes increases? In any case, the java
heap keeps steadily increasing, although it does drop when I invoke a manual
garbage collect.
Any ideas would be appreciated.
thanks
Andrew
--
View this message in context: http://www.nabble.com/References-to-objects-processed-in-routes-held%2C-eventually-run-out-of-memory-tp22580791p22588947.html
Sent from the Camel - Users mailing list archive at Nabble.com.