You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@servicemix.apache.org by Michal <ca...@yahoo.com> on 2006/12/30 09:22:51 UTC
Re: SedaFlow - OutOfMemoryError
I found the cause:
I had DEBUG level for logging.
DeliveryChannelImpl.doSend is logging MessageExchangeImpl.
Xerces was creating lots of arrays of int.
I am enclosing stack trace from netbeans profiler:
http://www.nabble.com/file/5170/logging_message.png
When I increased logging level (to WARN) then the problem dissapeared.
I wonder why it didn't happen in ST flow - doSend method is also called in
this flow?
Maybe in ST flow doSend method is called much less freqently?
However, I still have problems:
Jsr181ExchangeProcessor.doProcess sets XMLstreamReader to message.
Creating xmlstreamreader allocates lots of arrays of char's and they were
not garbage collected (live objects are equal to totally allocated).
I am enclosing stack trace from netbeans profiler:
http://www.nabble.com/file/5171/create_xml_stream_reader.png
Again, why this problem does not occur in ST flow? In ST flow doProcess
method is also called.
To recapitulate:
I am using SM3.0. ActiveMQ is embedded in SM. I have 64M memory (for testing
purpose).
Problems occurs when I use Seda flow - ST flow does not reports such
problems.
Here is my scenario:
- I have a webservice (therefore you can see Jsr181 classes on the stack)
that is accessible via http connector
- webservice validates xml (sends message to validate component) and then
sends xml to jms (via message exchange mechanism).
After around 1000 WS calls I get outofmemoryerror.
Please advise
gnodet wrote:
>
> Given your figures, I agree with your conclusion that it seems
> there is a memory leak in SEDA flow :(
> Could you raise a JIRA and attach the configuration you used
> so that we can reproduce the problem ?
>
> On 10/4/06, Michal <ca...@yahoo.com> wrote:
>>
>> Hi,
>>
>> I was sending lots of messages to activemq queue via servicemix using
>> sendSync method - there were no cunsumers connected to the queue. After
>> some
>> time I run out of memory.
>>
>> Here are the details:
>>
>> a) servicemix 3.0 M2, activemq (in seperate process), sedaflow - out of
>> memory
>> b) servicemix 3.0 M2, activemq (in seperate process), stflow - out of
>> memory
>> c) servicemix 3.0, activemq (in seperate process), sedaflow - out of
>> memory
>> d) servicemix 3.0, activemq (in seperate process), stflow - NO PROBLEM
>> e) servicemix 3.0, activemq (embedded broker, persistent=true,
>> memorymanagerlimit), sedaflow - out of memory
>> f) servicemix 3.0, activemq (embedded broker, persistent=true,
>> memorymanagerlimit), stflow - NO PROBLEM
>>
>> Conclusion:
>>
>> - there is a memory leak in M2 - indeed, I have seen som bugfix in 3.0
>> release so I used new version
>> - sedaflow eats up memory - memory leak?
>>
>> Is there really a memory leak in SedaFlow? Or maybe I need to configure
>> it
>> properly?
>> Is StFlow recommended for production?
>>
>>
>> Please advise.
>> Michal
>>
>>
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://www.nabble.com/SedaFlow---OutOfMemoryError-tf2382645.html#a6640548
>> Sent from the ServiceMix - User mailing list archive at Nabble.com.
>>
>>
>
>
> --
> Cheers,
> Guillaume Nodet
>
>
--
View this message in context: http://www.nabble.com/SedaFlow---OutOfMemoryError-tf2382645s12049.html#a8096425
Sent from the ServiceMix - User mailing list archive at Nabble.com.
Re: SedaFlow - OutOfMemoryError
Posted by Michal <ca...@yahoo.com>.
I have found the root cause of the problem -
org.apache.servicemix.http.processors.ConsumerProcessor has a memory leak.
Environment:
=========
I am using FUSE distribution version 3.2.0.0
Testing scenario:
==========
webserviceclient -> http SU -> jsr181 SU -> jms SU
I was sending lots of messages with WS client. When I was using ST flow then
SM memory usage was stable. However, with SEDA flow it was increasing and
finaly (with 128M) OOM error was thrown.
I have profiled the application and found that ConsumerProcessor holds
references to exchanges in its map. The problem is that an exchange is put
to the map (line 99) but the removal is risky (line 159). Using ST flow
(synchronous) there is a guarantee that 'put' will be called before
'remove'. However, in asynchronous flows (f.e. SEDA) there is no such
guarantee.
I think, to fix it an exchange would need to be removed right after
suspension is finished (line 163)
Anyone disagree with my conclusions? If not I will raise an issue on JIRA.
Michal
gnodet wrote:
>
> I would recommend to create a test case to isolate the xml
> parsing related problems. It seems the SourceTransformer is
> always in the stack trace, so it should be easy to write
> some simple tests to check for OOM. If you can write
> some tests that throws OOM, then try to upgrade xalan
> and xerces to the latest version, as it may solve the problem.
>
> On 12/30/06, Michal <ca...@yahoo.com> wrote:
>>
>> I found the cause:
>> I had DEBUG level for logging.
>> DeliveryChannelImpl.doSend is logging MessageExchangeImpl.
>> Xerces was creating lots of arrays of int.
>> I am enclosing stack trace from netbeans profiler:
>> http://www.nabble.com/file/5170/logging_message.png
>>
>> When I increased logging level (to WARN) then the problem dissapeared.
>> I wonder why it didn't happen in ST flow - doSend method is also called
>> in
>> this flow?
>> Maybe in ST flow doSend method is called much less freqently?
>>
>> However, I still have problems:
>>
>> Jsr181ExchangeProcessor.doProcess sets XMLstreamReader to message.
>> Creating xmlstreamreader allocates lots of arrays of char's and they were
>> not garbage collected (live objects are equal to totally allocated).
>> I am enclosing stack trace from netbeans profiler:
>> http://www.nabble.com/file/5171/create_xml_stream_reader.png
>>
>>
>> Again, why this problem does not occur in ST flow? In ST flow doProcess
>> method is also called.
>>
>> To recapitulate:
>> I am using SM3.0. ActiveMQ is embedded in SM. I have 64M memory (for
>> testing
>> purpose).
>> Problems occurs when I use Seda flow - ST flow does not reports such
>> problems.
>> Here is my scenario:
>> - I have a webservice (therefore you can see Jsr181 classes on the stack)
>> that is accessible via http connector
>> - webservice validates xml (sends message to validate component) and then
>> sends xml to jms (via message exchange mechanism).
>> After around 1000 WS calls I get outofmemoryerror.
>>
>> Please advise
>>
>>
>>
>> gnodet wrote:
>> >
>> > Given your figures, I agree with your conclusion that it seems
>> > there is a memory leak in SEDA flow :(
>> > Could you raise a JIRA and attach the configuration you used
>> > so that we can reproduce the problem ?
>> >
>> > On 10/4/06, Michal <ca...@yahoo.com> wrote:
>> >>
>> >> Hi,
>> >>
>> >> I was sending lots of messages to activemq queue via servicemix using
>> >> sendSync method - there were no cunsumers connected to the queue.
>> After
>> >> some
>> >> time I run out of memory.
>> >>
>> >> Here are the details:
>> >>
>> >> a) servicemix 3.0 M2, activemq (in seperate process), sedaflow - out
>> of
>> >> memory
>> >> b) servicemix 3.0 M2, activemq (in seperate process), stflow - out of
>> >> memory
>> >> c) servicemix 3.0, activemq (in seperate process), sedaflow - out of
>> >> memory
>> >> d) servicemix 3.0, activemq (in seperate process), stflow - NO PROBLEM
>> >> e) servicemix 3.0, activemq (embedded broker, persistent=true,
>> >> memorymanagerlimit), sedaflow - out of memory
>> >> f) servicemix 3.0, activemq (embedded broker, persistent=true,
>> >> memorymanagerlimit), stflow - NO PROBLEM
>> >>
>> >> Conclusion:
>> >>
>> >> - there is a memory leak in M2 - indeed, I have seen som bugfix in 3.0
>> >> release so I used new version
>> >> - sedaflow eats up memory - memory leak?
>> >>
>> >> Is there really a memory leak in SedaFlow? Or maybe I need to
>> configure
>> >> it
>> >> properly?
>> >> Is StFlow recommended for production?
>> >>
>> >>
>> >> Please advise.
>> >> Michal
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> --
>> >> View this message in context:
>> >>
>> http://www.nabble.com/SedaFlow---OutOfMemoryError-tf2382645.html#a6640548
>> >> Sent from the ServiceMix - User mailing list archive at Nabble.com.
>> >>
>> >>
>> >
>> >
>> > --
>> > Cheers,
>> > Guillaume Nodet
>> >
>> >
>>
>> --
>> View this message in context:
>> http://www.nabble.com/SedaFlow---OutOfMemoryError-tf2382645s12049.html#a8096425
>> Sent from the ServiceMix - User mailing list archive at Nabble.com.
>>
>>
>
>
> --
> Cheers,
> Guillaume Nodet
>
>
--
View this message in context: http://www.nabble.com/SedaFlow---OutOfMemoryError-tf2382645s12049.html#a11962112
Sent from the ServiceMix - User mailing list archive at Nabble.com.
Re: SedaFlow - OutOfMemoryError
Posted by Guillaume Nodet <gn...@gmail.com>.
I would recommend to create a test case to isolate the xml
parsing related problems. It seems the SourceTransformer is
always in the stack trace, so it should be easy to write
some simple tests to check for OOM. If you can write
some tests that throws OOM, then try to upgrade xalan
and xerces to the latest version, as it may solve the problem.
On 12/30/06, Michal <ca...@yahoo.com> wrote:
>
> I found the cause:
> I had DEBUG level for logging.
> DeliveryChannelImpl.doSend is logging MessageExchangeImpl.
> Xerces was creating lots of arrays of int.
> I am enclosing stack trace from netbeans profiler:
> http://www.nabble.com/file/5170/logging_message.png
>
> When I increased logging level (to WARN) then the problem dissapeared.
> I wonder why it didn't happen in ST flow - doSend method is also called in
> this flow?
> Maybe in ST flow doSend method is called much less freqently?
>
> However, I still have problems:
>
> Jsr181ExchangeProcessor.doProcess sets XMLstreamReader to message.
> Creating xmlstreamreader allocates lots of arrays of char's and they were
> not garbage collected (live objects are equal to totally allocated).
> I am enclosing stack trace from netbeans profiler:
> http://www.nabble.com/file/5171/create_xml_stream_reader.png
>
>
> Again, why this problem does not occur in ST flow? In ST flow doProcess
> method is also called.
>
> To recapitulate:
> I am using SM3.0. ActiveMQ is embedded in SM. I have 64M memory (for testing
> purpose).
> Problems occurs when I use Seda flow - ST flow does not reports such
> problems.
> Here is my scenario:
> - I have a webservice (therefore you can see Jsr181 classes on the stack)
> that is accessible via http connector
> - webservice validates xml (sends message to validate component) and then
> sends xml to jms (via message exchange mechanism).
> After around 1000 WS calls I get outofmemoryerror.
>
> Please advise
>
>
>
> gnodet wrote:
> >
> > Given your figures, I agree with your conclusion that it seems
> > there is a memory leak in SEDA flow :(
> > Could you raise a JIRA and attach the configuration you used
> > so that we can reproduce the problem ?
> >
> > On 10/4/06, Michal <ca...@yahoo.com> wrote:
> >>
> >> Hi,
> >>
> >> I was sending lots of messages to activemq queue via servicemix using
> >> sendSync method - there were no cunsumers connected to the queue. After
> >> some
> >> time I run out of memory.
> >>
> >> Here are the details:
> >>
> >> a) servicemix 3.0 M2, activemq (in seperate process), sedaflow - out of
> >> memory
> >> b) servicemix 3.0 M2, activemq (in seperate process), stflow - out of
> >> memory
> >> c) servicemix 3.0, activemq (in seperate process), sedaflow - out of
> >> memory
> >> d) servicemix 3.0, activemq (in seperate process), stflow - NO PROBLEM
> >> e) servicemix 3.0, activemq (embedded broker, persistent=true,
> >> memorymanagerlimit), sedaflow - out of memory
> >> f) servicemix 3.0, activemq (embedded broker, persistent=true,
> >> memorymanagerlimit), stflow - NO PROBLEM
> >>
> >> Conclusion:
> >>
> >> - there is a memory leak in M2 - indeed, I have seen som bugfix in 3.0
> >> release so I used new version
> >> - sedaflow eats up memory - memory leak?
> >>
> >> Is there really a memory leak in SedaFlow? Or maybe I need to configure
> >> it
> >> properly?
> >> Is StFlow recommended for production?
> >>
> >>
> >> Please advise.
> >> Michal
> >>
> >>
> >>
> >>
> >>
> >>
> >> --
> >> View this message in context:
> >> http://www.nabble.com/SedaFlow---OutOfMemoryError-tf2382645.html#a6640548
> >> Sent from the ServiceMix - User mailing list archive at Nabble.com.
> >>
> >>
> >
> >
> > --
> > Cheers,
> > Guillaume Nodet
> >
> >
>
> --
> View this message in context: http://www.nabble.com/SedaFlow---OutOfMemoryError-tf2382645s12049.html#a8096425
> Sent from the ServiceMix - User mailing list archive at Nabble.com.
>
>
--
Cheers,
Guillaume Nodet