You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Claus Ibsen <cl...@gmail.com> on 2009/03/19 14:29:13 UTC
Re: References to objects processed in routes held, eventually run
out of memory
Hi
Just a very quick reply without digging into it yet. There was/is a
leak in camel-jaxb.
I cant remember if we got it fixed in 1.6.0 or its in 1.6.1. At least
its fixed in 2.0m1.
Could you either:
- try without any JAXB
- try with 1.6.1-SNAPSHOT or 2.0m1/2.0-SNAPSHOT
On Thu, Mar 19, 2009 at 2:24 PM, ee7arh <an...@2e-systems.com> wrote:
>
> Hi,
>
> Thanks for response.
>
> I'm using Camel 1.6.0 and ActiveMQ 5.2.0. I also had the same results with
> camel 1.5 and activeMQ 5.1
>
> I have attached a graph from Jconsole showing what happens as the
> application starts, runs for a about 1.5 hours then crashes with out of
> memory. http://www.nabble.com/file/p22600122/jconsole-graphs.jpg
>
> Notes:
>
> - I set the maximum memory to 64MB using -Xmx64M -server. normally I would
> use more but then the problem takes longer to reproduce
> - Notice the number of loaded classes never reduces. I also have another
> application which shows the same pattern.
> - As the application starts to run out of memory, the CPU usage goes
> extremely high
>
> Here is my route:
>
> from("jms:queue:mobileExtTriggerInvites")
>
> .errorHandler(deadLetterChannel("seda:errors").maximumRedeliveries(0))
>
> // Exception Handling
> .onException(JAXBException.class)
> .to("jms:queue:unmarshallableRequest").end()
> .onException(UnexpectedEventException.class)
> .to("jms:queue:unexpectedEvent").end()
>
> // Log Incoming XML
> .to("log:incomingEventXmlLogger?level=INFO")
>
> // Attempt to Unmarshall object expected on this queue
> .to("bean:eventMarshaller?methodName=unmarshallTriggerInvites")
>
> // Generate Service Events from this External Event
>
> .to("bean:serviceEventGenerator?methodName=procecssTriggerInvites")
>
> // Split generated Events out of List into individual events
> .splitter(body())
>
> // Destination queue for further processing
> .to("jms:queue:serviceEventQueue");
>
> You can see that I have 2 bean processors, I have also attached the code for
> those.
>
> http://www.nabble.com/file/p22600122/EventMarhaller.txt EventMarhaller.txt
> http://www.nabble.com/file/p22600122/ServiceEventGenerator.txt
> ServiceEventGenerator.txt
>
> Perhaps it's useful, perhaps not but I read a blog article which sounded
> very similar to the problem I am having, link below. Basically it looks like
> Camel or ActiveMQ still has a reference to every single object I create and
> send through the routing - this would surely explain why the number of
> loaded classes gets bigger and bigger the more the application runs.
>
> http://dertompson.com/2007/08/01/fighting-the-outofmemoryerror/
> http://dertompson.com/2007/08/01/fighting-the-outofmemoryerror
>
> using JConsole, I also periodically purged the end queue and this made no
> difference.
>
> If it helps, I am able to provide a heapdump to anyone for analysis with
> jHat. As mentioned in my 1st post, all instances of classes created during
> the application lifecycle continue to live on in memory. It appears that my
> objects are referenced by ActiveMQMessage
>
> For example, I will trace 1 of those classes created in code attached:
>
> Class:
> class
> com.ee.berbe.mobile.servicelogic.flightops.events.service.TriggerMMSInvitations
>
> References to this object:
> org.apache.activemq.command.ActiveMQObjectMessage@0xafab8fd8 (174 bytes) :
> field object
>
> --> -->
>
> Class:
> class org.apache.activemq.command.ActiveMQObjectMessage
>
> References to this object:
> java.util.LinkedHashMap$Entry@0xafab8970 (32 bytes) : field value
> java.util.LinkedHashMap$Entry@0xafab8810 (32 bytes) : field value
>
>
> So it seems that ActiveMQ is still storing all my objets on a LinkedHashMap
> but I've no idea why.
>
> Thanks again for any help
>
> Andrew
>
>
> willem.jiang wrote:
>>
>> Hi
>> Which version of Camel are you using ?
>> Can you show us you routing rules and your processor codes?
>>
>> They will help us to dig the issue.
>>
>> Willem
>>
>>
> --
> View this message in context: http://www.nabble.com/References-to-objects-processed-in-routes-held%2C-eventually-run-out-of-memory-tp22580791p22600122.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>
--
Claus Ibsen
Apache Camel Committer
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Re: References to objects processed in routes held, eventually run
out of memory
Posted by ee7arh <an...@2e-systems.com>.
Hi again,
A bit more information which might help. I have been running the application
for the last 4 hours with 256MB memory and so far it looks like the
outOfMemory problem will probably not occur anytime soon because the heap
usage drops dramatically after the initial high load processing.
In addition, I have been running messages through further routes to perform
more processing. In my last example, i ran the applciation only through a
simple route but using less memory
Here is the graph:
http://www.nabble.com/file/p22601000/jconsole-graphs-2.jpg
When the application was first started it was under high load so you can see
it took almsot 200MB but after the first 30 mns, fewer messages were
processed and so the application looks to gradually recover the memory.
however, having said all that, notice once again the classes loaded into
memory, it does not follow the same pattern. Number of loaded classes keeps
on increasing so I guess although it may now take a long time, the
application will eventually run out of memory.
Perhaps then there are 2 reasons which can cause an out of memory:
1) When I simply do not have enough memory to facilitate high-load procesing
2) When the nunber of classes in memory continues to increase until no more
memory is left...
Andrew
ee7arh wrote:
>
> Hi,
>
> Thanks but we are not using camel-jaxb, everything is being done manually
> using the JaxB classes directly (although now I see camel has this built
> in I will change.... well for R2.0 anyway!)
>
> As way of double checking, I found only 7 instances of my unmarshalled
> objects in the memory dump, not 1600 like I see for the aforementioned
> object in previous post.
>
> Andrew
>
>
> Claus Ibsen-2 wrote:
>>
>> Hi
>>
>> Just a very quick reply without digging into it yet. There was/is a
>> leak in camel-jaxb.
>> I cant remember if we got it fixed in 1.6.0 or its in 1.6.1. At least
>> its fixed in 2.0m1.
>>
>> Could you either:
>> - try without any JAXB
>> - try with 1.6.1-SNAPSHOT or 2.0m1/2.0-SNAPSHOT
>>
>>
>>
>> On Thu, Mar 19, 2009 at 2:24 PM, ee7arh <an...@2e-systems.com>
>> wrote:
>>>
>>> Hi,
>>>
>>> Thanks for response.
>>>
>>> I'm using Camel 1.6.0 and ActiveMQ 5.2.0. I also had the same results
>>> with
>>> camel 1.5 and activeMQ 5.1
>>>
>>> I have attached a graph from Jconsole showing what happens as the
>>> application starts, runs for a about 1.5 hours then crashes with out of
>>> memory. http://www.nabble.com/file/p22600122/jconsole-graphs.jpg
>>>
>>> Notes:
>>>
>>> - I set the maximum memory to 64MB using -Xmx64M -server. normally I
>>> would
>>> use more but then the problem takes longer to reproduce
>>> - Notice the number of loaded classes never reduces. I also have another
>>> application which shows the same pattern.
>>> - As the application starts to run out of memory, the CPU usage goes
>>> extremely high
>>>
>>> Here is my route:
>>>
>>> from("jms:queue:mobileExtTriggerInvites")
>>>
>>> .errorHandler(deadLetterChannel("seda:errors").maximumRedeliveries(0))
>>>
>>> // Exception Handling
>>> .onException(JAXBException.class)
>>> .to("jms:queue:unmarshallableRequest").end()
>>> .onException(UnexpectedEventException.class)
>>> .to("jms:queue:unexpectedEvent").end()
>>>
>>> // Log Incoming XML
>>> .to("log:incomingEventXmlLogger?level=INFO")
>>>
>>> // Attempt to Unmarshall object expected on this queue
>>>
>>> .to("bean:eventMarshaller?methodName=unmarshallTriggerInvites")
>>>
>>> // Generate Service Events from this External Event
>>>
>>> .to("bean:serviceEventGenerator?methodName=procecssTriggerInvites")
>>>
>>> // Split generated Events out of List into individual events
>>> .splitter(body())
>>>
>>> // Destination queue for further processing
>>> .to("jms:queue:serviceEventQueue");
>>>
>>> You can see that I have 2 bean processors, I have also attached the code
>>> for
>>> those.
>>>
>>> http://www.nabble.com/file/p22600122/EventMarhaller.txt
>>> EventMarhaller.txt
>>> http://www.nabble.com/file/p22600122/ServiceEventGenerator.txt
>>> ServiceEventGenerator.txt
>>>
>>> Perhaps it's useful, perhaps not but I read a blog article which sounded
>>> very similar to the problem I am having, link below. Basically it looks
>>> like
>>> Camel or ActiveMQ still has a reference to every single object I create
>>> and
>>> send through the routing - this would surely explain why the number of
>>> loaded classes gets bigger and bigger the more the application runs.
>>>
>>> http://dertompson.com/2007/08/01/fighting-the-outofmemoryerror/
>>> http://dertompson.com/2007/08/01/fighting-the-outofmemoryerror
>>>
>>> using JConsole, I also periodically purged the end queue and this made
>>> no
>>> difference.
>>>
>>> If it helps, I am able to provide a heapdump to anyone for analysis with
>>> jHat. As mentioned in my 1st post, all instances of classes created
>>> during
>>> the application lifecycle continue to live on in memory. It appears that
>>> my
>>> objects are referenced by ActiveMQMessage
>>>
>>> For example, I will trace 1 of those classes created in code attached:
>>>
>>> Class:
>>> class
>>> com.ee.berbe.mobile.servicelogic.flightops.events.service.TriggerMMSInvitations
>>>
>>> References to this object:
>>> org.apache.activemq.command.ActiveMQObjectMessage@0xafab8fd8 (174 bytes)
>>> :
>>> field object
>>>
>>> --> -->
>>>
>>> Class:
>>> class org.apache.activemq.command.ActiveMQObjectMessage
>>>
>>> References to this object:
>>> java.util.LinkedHashMap$Entry@0xafab8970 (32 bytes) : field value
>>> java.util.LinkedHashMap$Entry@0xafab8810 (32 bytes) : field value
>>>
>>>
>>> So it seems that ActiveMQ is still storing all my objets on a
>>> LinkedHashMap
>>> but I've no idea why.
>>>
>>> Thanks again for any help
>>>
>>> Andrew
>>>
>>>
>>> willem.jiang wrote:
>>>>
>>>> Hi
>>>> Which version of Camel are you using ?
>>>> Can you show us you routing rules and your processor codes?
>>>>
>>>> They will help us to dig the issue.
>>>>
>>>> Willem
>>>>
>>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/References-to-objects-processed-in-routes-held%2C-eventually-run-out-of-memory-tp22580791p22600122.html
>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>>
>> --
>> Claus Ibsen
>> Apache Camel Committer
>>
>> Open Source Integration: http://fusesource.com
>> Blog: http://davsclaus.blogspot.com/
>>
>>
>
>
--
View this message in context: http://www.nabble.com/References-to-objects-processed-in-routes-held%2C-eventually-run-out-of-memory-tp22580791p22601000.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: References to objects processed in routes held, eventually run
out of memory
Posted by ee7arh <an...@2e-systems.com>.
Hi,
Thanks but we are not using camel-jaxb, everything is being done manually
using the JaxB classes directly (although now I see camel has this built in
I will change.... well for R2.0 anyway!)
As way of double checking, I found only 7 instances of my unmarshalled
objects in the memory dump, not 1600 like I see for the aforementioned
object in previous post.
Andrew
Claus Ibsen-2 wrote:
>
> Hi
>
> Just a very quick reply without digging into it yet. There was/is a
> leak in camel-jaxb.
> I cant remember if we got it fixed in 1.6.0 or its in 1.6.1. At least
> its fixed in 2.0m1.
>
> Could you either:
> - try without any JAXB
> - try with 1.6.1-SNAPSHOT or 2.0m1/2.0-SNAPSHOT
>
>
>
> On Thu, Mar 19, 2009 at 2:24 PM, ee7arh <an...@2e-systems.com>
> wrote:
>>
>> Hi,
>>
>> Thanks for response.
>>
>> I'm using Camel 1.6.0 and ActiveMQ 5.2.0. I also had the same results
>> with
>> camel 1.5 and activeMQ 5.1
>>
>> I have attached a graph from Jconsole showing what happens as the
>> application starts, runs for a about 1.5 hours then crashes with out of
>> memory. http://www.nabble.com/file/p22600122/jconsole-graphs.jpg
>>
>> Notes:
>>
>> - I set the maximum memory to 64MB using -Xmx64M -server. normally I
>> would
>> use more but then the problem takes longer to reproduce
>> - Notice the number of loaded classes never reduces. I also have another
>> application which shows the same pattern.
>> - As the application starts to run out of memory, the CPU usage goes
>> extremely high
>>
>> Here is my route:
>>
>> from("jms:queue:mobileExtTriggerInvites")
>>
>> .errorHandler(deadLetterChannel("seda:errors").maximumRedeliveries(0))
>>
>> // Exception Handling
>> .onException(JAXBException.class)
>> .to("jms:queue:unmarshallableRequest").end()
>> .onException(UnexpectedEventException.class)
>> .to("jms:queue:unexpectedEvent").end()
>>
>> // Log Incoming XML
>> .to("log:incomingEventXmlLogger?level=INFO")
>>
>> // Attempt to Unmarshall object expected on this queue
>>
>> .to("bean:eventMarshaller?methodName=unmarshallTriggerInvites")
>>
>> // Generate Service Events from this External Event
>>
>> .to("bean:serviceEventGenerator?methodName=procecssTriggerInvites")
>>
>> // Split generated Events out of List into individual events
>> .splitter(body())
>>
>> // Destination queue for further processing
>> .to("jms:queue:serviceEventQueue");
>>
>> You can see that I have 2 bean processors, I have also attached the code
>> for
>> those.
>>
>> http://www.nabble.com/file/p22600122/EventMarhaller.txt
>> EventMarhaller.txt
>> http://www.nabble.com/file/p22600122/ServiceEventGenerator.txt
>> ServiceEventGenerator.txt
>>
>> Perhaps it's useful, perhaps not but I read a blog article which sounded
>> very similar to the problem I am having, link below. Basically it looks
>> like
>> Camel or ActiveMQ still has a reference to every single object I create
>> and
>> send through the routing - this would surely explain why the number of
>> loaded classes gets bigger and bigger the more the application runs.
>>
>> http://dertompson.com/2007/08/01/fighting-the-outofmemoryerror/
>> http://dertompson.com/2007/08/01/fighting-the-outofmemoryerror
>>
>> using JConsole, I also periodically purged the end queue and this made no
>> difference.
>>
>> If it helps, I am able to provide a heapdump to anyone for analysis with
>> jHat. As mentioned in my 1st post, all instances of classes created
>> during
>> the application lifecycle continue to live on in memory. It appears that
>> my
>> objects are referenced by ActiveMQMessage
>>
>> For example, I will trace 1 of those classes created in code attached:
>>
>> Class:
>> class
>> com.ee.berbe.mobile.servicelogic.flightops.events.service.TriggerMMSInvitations
>>
>> References to this object:
>> org.apache.activemq.command.ActiveMQObjectMessage@0xafab8fd8 (174 bytes)
>> :
>> field object
>>
>> --> -->
>>
>> Class:
>> class org.apache.activemq.command.ActiveMQObjectMessage
>>
>> References to this object:
>> java.util.LinkedHashMap$Entry@0xafab8970 (32 bytes) : field value
>> java.util.LinkedHashMap$Entry@0xafab8810 (32 bytes) : field value
>>
>>
>> So it seems that ActiveMQ is still storing all my objets on a
>> LinkedHashMap
>> but I've no idea why.
>>
>> Thanks again for any help
>>
>> Andrew
>>
>>
>> willem.jiang wrote:
>>>
>>> Hi
>>> Which version of Camel are you using ?
>>> Can you show us you routing rules and your processor codes?
>>>
>>> They will help us to dig the issue.
>>>
>>> Willem
>>>
>>>
>> --
>> View this message in context:
>> http://www.nabble.com/References-to-objects-processed-in-routes-held%2C-eventually-run-out-of-memory-tp22580791p22600122.html
>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>
>>
>
>
>
> --
> Claus Ibsen
> Apache Camel Committer
>
> Open Source Integration: http://fusesource.com
> Blog: http://davsclaus.blogspot.com/
>
>
--
View this message in context: http://www.nabble.com/References-to-objects-processed-in-routes-held%2C-eventually-run-out-of-memory-tp22580791p22600424.html
Sent from the Camel - Users mailing list archive at Nabble.com.