You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@servicemix.apache.org by Stuart Roe <st...@ovic.co.uk> on 2009/11/20 15:46:13 UTC
Not deallocating resources causing Old Gen memory leaks
Hi all,
I appear to have a memory leak while performing an exchange and I'm unsure
how to identify it/correct it. Bascially, my application happily runs for a
few days and then fails with an out of memory exception. Using jconsole I
can see that the memory usage is bouncing around with a slow consistent
creap up. This is why I suspect a memory leak.
The message path within the app is cxf-bc (ver. 2009.01) -> cxf-se (ver.
2009.01) all running in SMX 3.3.1/WinXP.
The handler in CXF-SE creates a new InOut message and sync. posts it to the
NMR, unmarshals the response into a reply. There is a copy of the handler
function below. I have removed the actual data handling and exceptions.
My question is, I'm I missing some form of deallocation/close call that will
return/release an object and hence remove my memory leak?
I have found that replacing the message exchange from the handler and hard
wiring a response fixes the leak.
Any help/pointers would be helpful.
Stuart.
BTW:
I'm using JiBX to marshal between NMR XML and Java objects (which may also
be the source of the problem).
public List<TypeConfigStatus> getConfigStatus()
try {
GetConfigStatusResponse response = null;
DeliveryChannel ch = getContext().getDeliveryChannel();
InOut exch = ch.createExchangeFactory().createInOutExchange();
exch.setService(getService());
exch.setInMessage(exch.createMessage());
exch.getInMessage().setProperty("operation", "config-status");
exch.getInMessage().setContent(new StringSource("<empty />"));
if(ch.sendSync(exch)){
NormalizedMessage amsg = exch.getOutMessage();
try {
SourceTransformer trans = new SourceTransformer();
String content = trans.contentToString(amsg);
IBindingFactory bfact =
BindingDirectory.getFactory(GetConfigStatusResponse.class);
IUnmarshallingContext uctx = bfact.createUnmarshallingContext();
response = (GetConfigStatusResponse) uctx.unmarshalDocument(new
StringReader(content));
} catch
// block removed
}finally{
exch.setStatus(ExchangeStatus.DONE);
ch.send(exch);
}
}
// set result = cooked(response)
return result;
}
Re: Not deallocating resources causing Old Gen memory leaks
Posted by Gert Vanthienen <ge...@gmail.com>.
Stuart,
I added a new test case to the SMXCOMP-24 issue, now using a consumer
bean to start the InOut MessageExchange, but still no luck in
reproducing the issue. Both tests in the test case seem to pass. Is
this the scenario you had in mind?
Btw, in the case you mentioned, the response sent in step 2 will not
be removed from the map at that time, but when the DONE exchange is
sent in step 3 from the consumer to the provider, the DONE
MessageExchange is being received by the provider endpoint and will be
removed at that time.
Regards,
Gert Vanthienen
------------------------
Open Source SOA: http://fusesource.com
Blog: http://gertvanthienen.blogspot.com/
2009/12/1 Stuart Roe <st...@ovic.co.uk>:
> Gert,
>
> Thinking about it I'm not convinced that it is the same issue, just related.
>
> Try creating a test that:
> 1). Consumer creates and send/sendSync an InOut message.
> 2). Provider (implemented via MessageExchangeListener) replies via send.
> 3). Consumer replies done.
>
> I think that the send in step 2 places a copy of the exchange into a
> HashMap. As the exchange is still active, it isn't removed by
> checkEndOfRequest(Request) and needs removing at a later point in time. I
> don't think that this is happening and hence the build up of
> org.apache.servicemix.jbi.messaging objects.
>
> If sendSync is used in replacement of send, then the exchange completes in
> sequence so the exchange is removed from the HashMap by
> checkEndOfRequest(Request) and the org.apache.servicemix.jbi.messaging
> objects don't collect.
>
> I don't think that using send on exchanges that state = done is a problem as
> the exchange is no active and checkEndOfRequest(Request) removes it from the
> HashMap.
>
> BTW: The bug report refers to SMX3.2.2, I'm working on SMX3.3.1,
> servicemix-bean v2009.01
>
> Stuart.
>
> -----Original Message-----
> From: Gert Vanthienen [mailto:gert.vanthienen@gmail.com]
> Sent: 30 November 2009 22:03
> To: users@servicemix.apache.org
> Subject: Re: Not deallocating resources causing Old Gen memory leaks
>
> Stuart,
>
> Before attempting to fix the issue, I would like to write a unit test that
> exposes the issue. From what I read here, is the issue being caused by the
> ExchangeStatus.DONE being sent with a plain send() (i.e.
> using sendSync() in the finally block fixes it?) ?
>
> I have added the test I was writing to the SMXCOMP-24 issue but that test
> succeeds so I must have missed something there. Can you help me pinpoint
> what part of the issue I've been misinterpreting?
>
> Regards,
>
> Gert Vanthienen
> ------------------------
> Open Source SOA: http://fusesource.com
> Blog: http://gertvanthienen.blogspot.com/
>
>
>
> 2009/11/30 Stuart Roe <st...@ovic.co.uk>:
>> Guys,
>>
>> FYI:
>> After some investigation, I think that this issue JIRA issue SMXCOMP-24.
>> Replacing the async reply with a sync reply appears to fix the leak,
>> although I think that is an other leak within activeMQ or within the
>> handling of activeMQ. The memory usage graphs appear to show an
>> increase (although significantly slower - so it may not be a leak!).
>> The classes witha a large number of instances are:
>>
>> java.util.HashMap$Entry
>> org.apache.activemq.command.ConsumerId
>> org.apache.activemq.command.SessionId
>> org.apache.activemq.command.ConnectionId
>>
>> Stuart.
>>
>>
>> -----Original Message-----
>> From: Stuart Roe [mailto:stuart.roe@ovic.co.uk]
>> Sent: 24 November 2009 11:28
>> To: users@servicemix.apache.org
>> Subject: RE: Not deallocating resources causing Old Gen memory leaks
>>
>> Gert,
>>
>> Thanks for the reply. I have taken your advise and used jmap's histo
>> option to collect information on the number of instances of objects. I
>> ran a test lastnight and have taken numerous histograms (jmap -histo)
>> this morning and have identified (I think) a memory leak in the handling
> of message exchange.
>>
>>
>> The histograms were created by hitting the GC button in jconsole and
>> running the jmap command straight afterwards (trying to reduce the
>> noise from other normal create->use->destroy operations). The
>> following list shows the number of instances for 6 samples followed by
>> the name of the class. Unfortunately, the period between the samples
>> has not been consistent, so it's not possible to calculte a leak rate.
>>
>> There is a lot of noise on the java.lang.String class, but all the
>> other classes show a consistent increase in the number of instances
>> and there is a large number of servicemix/jbi/messaging related classes
> mentioned.
>>
>> From this, I conclude that there is:
>> A). a memory leak within the SMX 3.3.1/eip 2009.01/bean 2009.01.
>> B). my code is missing a close/release/dispose function call releasing
>> the resources.
>>
>> Could I encounter a problem if the createExchangeFactory() function
>> was called every time a message was received? Should the returned
>> factory be cached for reuse?
>>
>> Stuart.
>>
>> -> Process runtime increasing ->
>> 26849 30911 39184 45096 52555 64386
>> java.util.HashMap$Entry
>> 23261 32837 44761 28405 34285 29189 java.lang.String
>> 10754 12122 15144 17748 20526 25229 java.util.HashMap
>> 10745 12156 15194 17690 20460 25090
>> [Ljava.util.HashMap$Entry;
>> 12327 14564 16157 16252 18389 22281 [I
>> 7268 8042 10048 12098 14030 17384
>> org.apache.servicemix.jbi.messaging.InOutImpl
>> 7268 8042 10048 12098 14030 17384
>> org.apache.servicemix.jbi.messaging.NormalizedMessageImpl
>> 7272 8047 10054 12100 14032 17381
>> org.apache.servicemix.jbi.jaxp.StringSource
>> 3636 4022 5024 6053 7019 8698
>> org.apache.servicemix.bean.support.Request
>> 3634 4021 5024 6049 7015 8692
>> org.apache.servicemix.jbi.messaging.MessageExchangeFactoryImpl$PrettyC
>> alenda
>> r
>> 3634 4021 5024 6049 7015 8692
>> sun.util.calendar.Gregorian$Date
>> 3634 4021 5024 6049 7015 8692
>> org.apache.servicemix.jbi.messaging.ExchangePacket
>> 3634 4024 5026 6046 7011 8685
>> java.util.HashMap$EntrySet
>> 3667 4098 5113 6030 6972 8578 [Z
>> 3624 4067 5080 5975 6890 8439
>> java.util.concurrent.ConcurrentHashMap$HashEntry
>> 3535 4005 5027 5875 6802 8352 java.util.HashSet
>>
>> -----Original Message-----
>> From: Gert Vanthienen [mailto:gert.vanthienen@gmail.com]
>> Sent: 23 November 2009 09:30
>> To: users@servicemix.apache.org
>> Subject: Re: Not deallocating resources causing Old Gen memory leaks
>>
>> L.S.,
>>
>> I'm not seeing anything obvious missing from this snippet of code.
>> Have you tried taking a memory dump and use jmap or yourkit or
>> something to find the objects that are causing the memore leak?
>>
>> Regards,
>>
>> Gert Vanthienen
>> ------------------------
>> Open Source SOA: http://fusesource.com
>> Blog: http://gertvanthienen.blogspot.com/
>>
>>
>>
>> 2009/11/20 Stuart Roe <st...@ovic.co.uk>:
>>> Hi all,
>>>
>>> I appear to have a memory leak while performing an exchange and I'm
>>> unsure how to identify it/correct it. Bascially, my application
>>> happily runs for a few days and then fails with an out of memory
>>> exception. Using jconsole I can see that the memory usage is bouncing
>>> around with a slow consistent creap up. This is why I suspect a
>>> memory
>> leak.
>>>
>>> The message path within the app is cxf-bc (ver. 2009.01) -> cxf-se (ver.
>>> 2009.01) all running in SMX 3.3.1/WinXP.
>>>
>>> The handler in CXF-SE creates a new InOut message and sync. posts it
>>> to the NMR, unmarshals the response into a reply. There is a copy of
>>> the handler function below. I have removed the actual data handling
>>> and
>> exceptions.
>>>
>>> My question is, I'm I missing some form of deallocation/close call
>>> that will return/release an object and hence remove my memory leak?
>>> I have found that replacing the message exchange from the handler and
>>> hard wiring a response fixes the leak.
>>>
>>> Any help/pointers would be helpful.
>>>
>>> Stuart.
>>>
>>> BTW:
>>> I'm using JiBX to marshal between NMR XML and Java objects (which may
>>> also be the source of the problem).
>>>
>>>
>>> public List<TypeConfigStatus> getConfigStatus()
>>> try {
>>> GetConfigStatusResponse response = null;
>>>
>>> DeliveryChannel ch = getContext().getDeliveryChannel();
>>> InOut exch = ch.createExchangeFactory().createInOutExchange();
>>> exch.setService(getService());
>>> exch.setInMessage(exch.createMessage());
>>> exch.getInMessage().setProperty("operation", "config-status");
>>> exch.getInMessage().setContent(new StringSource("<empty />"));
>>>
>>> if(ch.sendSync(exch)){
>>> NormalizedMessage amsg = exch.getOutMessage();
>>> try {
>>> SourceTransformer trans = new SourceTransformer();
>>> String content = trans.contentToString(amsg);
>>> IBindingFactory bfact =
>>> BindingDirectory.getFactory(GetConfigStatusResponse.class);
>>> IUnmarshallingContext uctx =
>>> bfact.createUnmarshallingContext();
>>> response = (GetConfigStatusResponse)
>>> uctx.unmarshalDocument(new StringReader(content));
>>>
>>> } catch
>>> // block removed
>>> }finally{
>>> exch.setStatus(ExchangeStatus.DONE);
>>> ch.send(exch);
>>> }
>>> }
>>>
>>> // set result = cooked(response)
>>>
>>> return result;
>>> }
>>>
>>>
>>
>>
>
>
RE: Not deallocating resources causing Old Gen memory leaks
Posted by Stuart Roe <st...@ovic.co.uk>.
Gert,
Thinking about it I'm not convinced that it is the same issue, just related.
Try creating a test that:
1). Consumer creates and send/sendSync an InOut message.
2). Provider (implemented via MessageExchangeListener) replies via send.
3). Consumer replies done.
I think that the send in step 2 places a copy of the exchange into a
HashMap. As the exchange is still active, it isn't removed by
checkEndOfRequest(Request) and needs removing at a later point in time. I
don't think that this is happening and hence the build up of
org.apache.servicemix.jbi.messaging objects.
If sendSync is used in replacement of send, then the exchange completes in
sequence so the exchange is removed from the HashMap by
checkEndOfRequest(Request) and the org.apache.servicemix.jbi.messaging
objects don't collect.
I don't think that using send on exchanges that state = done is a problem as
the exchange is no active and checkEndOfRequest(Request) removes it from the
HashMap.
BTW: The bug report refers to SMX3.2.2, I'm working on SMX3.3.1,
servicemix-bean v2009.01
Stuart.
-----Original Message-----
From: Gert Vanthienen [mailto:gert.vanthienen@gmail.com]
Sent: 30 November 2009 22:03
To: users@servicemix.apache.org
Subject: Re: Not deallocating resources causing Old Gen memory leaks
Stuart,
Before attempting to fix the issue, I would like to write a unit test that
exposes the issue. From what I read here, is the issue being caused by the
ExchangeStatus.DONE being sent with a plain send() (i.e.
using sendSync() in the finally block fixes it?) ?
I have added the test I was writing to the SMXCOMP-24 issue but that test
succeeds so I must have missed something there. Can you help me pinpoint
what part of the issue I've been misinterpreting?
Regards,
Gert Vanthienen
------------------------
Open Source SOA: http://fusesource.com
Blog: http://gertvanthienen.blogspot.com/
2009/11/30 Stuart Roe <st...@ovic.co.uk>:
> Guys,
>
> FYI:
> After some investigation, I think that this issue JIRA issue SMXCOMP-24.
> Replacing the async reply with a sync reply appears to fix the leak,
> although I think that is an other leak within activeMQ or within the
> handling of activeMQ. The memory usage graphs appear to show an
> increase (although significantly slower - so it may not be a leak!).
> The classes witha a large number of instances are:
>
> java.util.HashMap$Entry
> org.apache.activemq.command.ConsumerId
> org.apache.activemq.command.SessionId
> org.apache.activemq.command.ConnectionId
>
> Stuart.
>
>
> -----Original Message-----
> From: Stuart Roe [mailto:stuart.roe@ovic.co.uk]
> Sent: 24 November 2009 11:28
> To: users@servicemix.apache.org
> Subject: RE: Not deallocating resources causing Old Gen memory leaks
>
> Gert,
>
> Thanks for the reply. I have taken your advise and used jmap's histo
> option to collect information on the number of instances of objects. I
> ran a test lastnight and have taken numerous histograms (jmap -histo)
> this morning and have identified (I think) a memory leak in the handling
of message exchange.
>
>
> The histograms were created by hitting the GC button in jconsole and
> running the jmap command straight afterwards (trying to reduce the
> noise from other normal create->use->destroy operations). The
> following list shows the number of instances for 6 samples followed by
> the name of the class. Unfortunately, the period between the samples
> has not been consistent, so it's not possible to calculte a leak rate.
>
> There is a lot of noise on the java.lang.String class, but all the
> other classes show a consistent increase in the number of instances
> and there is a large number of servicemix/jbi/messaging related classes
mentioned.
>
> From this, I conclude that there is:
> A). a memory leak within the SMX 3.3.1/eip 2009.01/bean 2009.01.
> B). my code is missing a close/release/dispose function call releasing
> the resources.
>
> Could I encounter a problem if the createExchangeFactory() function
> was called every time a message was received? Should the returned
> factory be cached for reuse?
>
> Stuart.
>
> -> Process runtime increasing ->
> 26849 30911 39184 45096 52555 64386
> java.util.HashMap$Entry
> 23261 32837 44761 28405 34285 29189 java.lang.String
> 10754 12122 15144 17748 20526 25229 java.util.HashMap
> 10745 12156 15194 17690 20460 25090
> [Ljava.util.HashMap$Entry;
> 12327 14564 16157 16252 18389 22281 [I
> 7268 8042 10048 12098 14030 17384
> org.apache.servicemix.jbi.messaging.InOutImpl
> 7268 8042 10048 12098 14030 17384
> org.apache.servicemix.jbi.messaging.NormalizedMessageImpl
> 7272 8047 10054 12100 14032 17381
> org.apache.servicemix.jbi.jaxp.StringSource
> 3636 4022 5024 6053 7019 8698
> org.apache.servicemix.bean.support.Request
> 3634 4021 5024 6049 7015 8692
> org.apache.servicemix.jbi.messaging.MessageExchangeFactoryImpl$PrettyC
> alenda
> r
> 3634 4021 5024 6049 7015 8692
> sun.util.calendar.Gregorian$Date
> 3634 4021 5024 6049 7015 8692
> org.apache.servicemix.jbi.messaging.ExchangePacket
> 3634 4024 5026 6046 7011 8685
> java.util.HashMap$EntrySet
> 3667 4098 5113 6030 6972 8578 [Z
> 3624 4067 5080 5975 6890 8439
> java.util.concurrent.ConcurrentHashMap$HashEntry
> 3535 4005 5027 5875 6802 8352 java.util.HashSet
>
> -----Original Message-----
> From: Gert Vanthienen [mailto:gert.vanthienen@gmail.com]
> Sent: 23 November 2009 09:30
> To: users@servicemix.apache.org
> Subject: Re: Not deallocating resources causing Old Gen memory leaks
>
> L.S.,
>
> I'm not seeing anything obvious missing from this snippet of code.
> Have you tried taking a memory dump and use jmap or yourkit or
> something to find the objects that are causing the memore leak?
>
> Regards,
>
> Gert Vanthienen
> ------------------------
> Open Source SOA: http://fusesource.com
> Blog: http://gertvanthienen.blogspot.com/
>
>
>
> 2009/11/20 Stuart Roe <st...@ovic.co.uk>:
>> Hi all,
>>
>> I appear to have a memory leak while performing an exchange and I'm
>> unsure how to identify it/correct it. Bascially, my application
>> happily runs for a few days and then fails with an out of memory
>> exception. Using jconsole I can see that the memory usage is bouncing
>> around with a slow consistent creap up. This is why I suspect a
>> memory
> leak.
>>
>> The message path within the app is cxf-bc (ver. 2009.01) -> cxf-se (ver.
>> 2009.01) all running in SMX 3.3.1/WinXP.
>>
>> The handler in CXF-SE creates a new InOut message and sync. posts it
>> to the NMR, unmarshals the response into a reply. There is a copy of
>> the handler function below. I have removed the actual data handling
>> and
> exceptions.
>>
>> My question is, I'm I missing some form of deallocation/close call
>> that will return/release an object and hence remove my memory leak?
>> I have found that replacing the message exchange from the handler and
>> hard wiring a response fixes the leak.
>>
>> Any help/pointers would be helpful.
>>
>> Stuart.
>>
>> BTW:
>> I'm using JiBX to marshal between NMR XML and Java objects (which may
>> also be the source of the problem).
>>
>>
>> public List<TypeConfigStatus> getConfigStatus()
>> try {
>> GetConfigStatusResponse response = null;
>>
>> DeliveryChannel ch = getContext().getDeliveryChannel();
>> InOut exch = ch.createExchangeFactory().createInOutExchange();
>> exch.setService(getService());
>> exch.setInMessage(exch.createMessage());
>> exch.getInMessage().setProperty("operation", "config-status");
>> exch.getInMessage().setContent(new StringSource("<empty />"));
>>
>> if(ch.sendSync(exch)){
>> NormalizedMessage amsg = exch.getOutMessage();
>> try {
>> SourceTransformer trans = new SourceTransformer();
>> String content = trans.contentToString(amsg);
>> IBindingFactory bfact =
>> BindingDirectory.getFactory(GetConfigStatusResponse.class);
>> IUnmarshallingContext uctx =
>> bfact.createUnmarshallingContext();
>> response = (GetConfigStatusResponse)
>> uctx.unmarshalDocument(new StringReader(content));
>>
>> } catch
>> // block removed
>> }finally{
>> exch.setStatus(ExchangeStatus.DONE);
>> ch.send(exch);
>> }
>> }
>>
>> // set result = cooked(response)
>>
>> return result;
>> }
>>
>>
>
>
Re: Not deallocating resources causing Old Gen memory leaks
Posted by Gert Vanthienen <ge...@gmail.com>.
Stuart,
Before attempting to fix the issue, I would like to write a unit test
that exposes the issue. From what I read here, is the issue being
caused by the ExchangeStatus.DONE being sent with a plain send() (i.e.
using sendSync() in the finally block fixes it?) ?
I have added the test I was writing to the SMXCOMP-24 issue but that
test succeeds so I must have missed something there. Can you help me
pinpoint what part of the issue I've been misinterpreting?
Regards,
Gert Vanthienen
------------------------
Open Source SOA: http://fusesource.com
Blog: http://gertvanthienen.blogspot.com/
2009/11/30 Stuart Roe <st...@ovic.co.uk>:
> Guys,
>
> FYI:
> After some investigation, I think that this issue JIRA issue SMXCOMP-24.
> Replacing the async reply with a sync reply appears to fix the leak,
> although I think that is an other leak within activeMQ or within the
> handling of activeMQ. The memory usage graphs appear to show an increase
> (although significantly slower - so it may not be a leak!). The classes
> witha a large number of instances are:
>
> java.util.HashMap$Entry
> org.apache.activemq.command.ConsumerId
> org.apache.activemq.command.SessionId
> org.apache.activemq.command.ConnectionId
>
> Stuart.
>
>
> -----Original Message-----
> From: Stuart Roe [mailto:stuart.roe@ovic.co.uk]
> Sent: 24 November 2009 11:28
> To: users@servicemix.apache.org
> Subject: RE: Not deallocating resources causing Old Gen memory leaks
>
> Gert,
>
> Thanks for the reply. I have taken your advise and used jmap's histo option
> to collect information on the number of instances of objects. I ran a test
> lastnight and have taken numerous histograms (jmap -histo) this morning and
> have identified (I think) a memory leak in the handling of message exchange.
>
>
> The histograms were created by hitting the GC button in jconsole and running
> the jmap command straight afterwards (trying to reduce the noise from other
> normal create->use->destroy operations). The following list shows the number
> of instances for 6 samples followed by the name of the class. Unfortunately,
> the period between the samples has not been consistent, so it's not possible
> to calculte a leak rate.
>
> There is a lot of noise on the java.lang.String class, but all the other
> classes show a consistent increase in the number of instances and there is a
> large number of servicemix/jbi/messaging related classes mentioned.
>
> From this, I conclude that there is:
> A). a memory leak within the SMX 3.3.1/eip 2009.01/bean 2009.01.
> B). my code is missing a close/release/dispose function call releasing the
> resources.
>
> Could I encounter a problem if the createExchangeFactory() function was
> called every time a message was received? Should the returned factory be
> cached for reuse?
>
> Stuart.
>
> -> Process runtime increasing ->
> 26849 30911 39184 45096 52555 64386 java.util.HashMap$Entry
> 23261 32837 44761 28405 34285 29189 java.lang.String
> 10754 12122 15144 17748 20526 25229 java.util.HashMap
> 10745 12156 15194 17690 20460 25090 [Ljava.util.HashMap$Entry;
> 12327 14564 16157 16252 18389 22281 [I
> 7268 8042 10048 12098 14030 17384
> org.apache.servicemix.jbi.messaging.InOutImpl
> 7268 8042 10048 12098 14030 17384
> org.apache.servicemix.jbi.messaging.NormalizedMessageImpl
> 7272 8047 10054 12100 14032 17381
> org.apache.servicemix.jbi.jaxp.StringSource
> 3636 4022 5024 6053 7019 8698
> org.apache.servicemix.bean.support.Request
> 3634 4021 5024 6049 7015 8692
> org.apache.servicemix.jbi.messaging.MessageExchangeFactoryImpl$PrettyCalenda
> r
> 3634 4021 5024 6049 7015 8692
> sun.util.calendar.Gregorian$Date
> 3634 4021 5024 6049 7015 8692
> org.apache.servicemix.jbi.messaging.ExchangePacket
> 3634 4024 5026 6046 7011 8685 java.util.HashMap$EntrySet
> 3667 4098 5113 6030 6972 8578 [Z
> 3624 4067 5080 5975 6890 8439
> java.util.concurrent.ConcurrentHashMap$HashEntry
> 3535 4005 5027 5875 6802 8352 java.util.HashSet
>
> -----Original Message-----
> From: Gert Vanthienen [mailto:gert.vanthienen@gmail.com]
> Sent: 23 November 2009 09:30
> To: users@servicemix.apache.org
> Subject: Re: Not deallocating resources causing Old Gen memory leaks
>
> L.S.,
>
> I'm not seeing anything obvious missing from this snippet of code.
> Have you tried taking a memory dump and use jmap or yourkit or something to
> find the objects that are causing the memore leak?
>
> Regards,
>
> Gert Vanthienen
> ------------------------
> Open Source SOA: http://fusesource.com
> Blog: http://gertvanthienen.blogspot.com/
>
>
>
> 2009/11/20 Stuart Roe <st...@ovic.co.uk>:
>> Hi all,
>>
>> I appear to have a memory leak while performing an exchange and I'm
>> unsure how to identify it/correct it. Bascially, my application
>> happily runs for a few days and then fails with an out of memory
>> exception. Using jconsole I can see that the memory usage is bouncing
>> around with a slow consistent creap up. This is why I suspect a memory
> leak.
>>
>> The message path within the app is cxf-bc (ver. 2009.01) -> cxf-se (ver.
>> 2009.01) all running in SMX 3.3.1/WinXP.
>>
>> The handler in CXF-SE creates a new InOut message and sync. posts it
>> to the NMR, unmarshals the response into a reply. There is a copy of
>> the handler function below. I have removed the actual data handling
>> and
> exceptions.
>>
>> My question is, I'm I missing some form of deallocation/close call
>> that will return/release an object and hence remove my memory leak?
>> I have found that replacing the message exchange from the handler and
>> hard wiring a response fixes the leak.
>>
>> Any help/pointers would be helpful.
>>
>> Stuart.
>>
>> BTW:
>> I'm using JiBX to marshal between NMR XML and Java objects (which may
>> also be the source of the problem).
>>
>>
>> public List<TypeConfigStatus> getConfigStatus()
>> try {
>> GetConfigStatusResponse response = null;
>>
>> DeliveryChannel ch = getContext().getDeliveryChannel();
>> InOut exch = ch.createExchangeFactory().createInOutExchange();
>> exch.setService(getService());
>> exch.setInMessage(exch.createMessage());
>> exch.getInMessage().setProperty("operation", "config-status");
>> exch.getInMessage().setContent(new StringSource("<empty />"));
>>
>> if(ch.sendSync(exch)){
>> NormalizedMessage amsg = exch.getOutMessage();
>> try {
>> SourceTransformer trans = new SourceTransformer();
>> String content = trans.contentToString(amsg);
>> IBindingFactory bfact =
>> BindingDirectory.getFactory(GetConfigStatusResponse.class);
>> IUnmarshallingContext uctx =
>> bfact.createUnmarshallingContext();
>> response = (GetConfigStatusResponse)
>> uctx.unmarshalDocument(new StringReader(content));
>>
>> } catch
>> // block removed
>> }finally{
>> exch.setStatus(ExchangeStatus.DONE);
>> ch.send(exch);
>> }
>> }
>>
>> // set result = cooked(response)
>>
>> return result;
>> }
>>
>>
>
>
RE: Not deallocating resources causing Old Gen memory leaks
Posted by Stuart Roe <st...@ovic.co.uk>.
Guys,
FYI:
After some investigation, I think that this issue JIRA issue SMXCOMP-24.
Replacing the async reply with a sync reply appears to fix the leak,
although I think that is an other leak within activeMQ or within the
handling of activeMQ. The memory usage graphs appear to show an increase
(although significantly slower - so it may not be a leak!). The classes
witha a large number of instances are:
java.util.HashMap$Entry
org.apache.activemq.command.ConsumerId
org.apache.activemq.command.SessionId
org.apache.activemq.command.ConnectionId
Stuart.
-----Original Message-----
From: Stuart Roe [mailto:stuart.roe@ovic.co.uk]
Sent: 24 November 2009 11:28
To: users@servicemix.apache.org
Subject: RE: Not deallocating resources causing Old Gen memory leaks
Gert,
Thanks for the reply. I have taken your advise and used jmap's histo option
to collect information on the number of instances of objects. I ran a test
lastnight and have taken numerous histograms (jmap -histo) this morning and
have identified (I think) a memory leak in the handling of message exchange.
The histograms were created by hitting the GC button in jconsole and running
the jmap command straight afterwards (trying to reduce the noise from other
normal create->use->destroy operations). The following list shows the number
of instances for 6 samples followed by the name of the class. Unfortunately,
the period between the samples has not been consistent, so it's not possible
to calculte a leak rate.
There is a lot of noise on the java.lang.String class, but all the other
classes show a consistent increase in the number of instances and there is a
large number of servicemix/jbi/messaging related classes mentioned.
>From this, I conclude that there is:
A). a memory leak within the SMX 3.3.1/eip 2009.01/bean 2009.01.
B). my code is missing a close/release/dispose function call releasing the
resources.
Could I encounter a problem if the createExchangeFactory() function was
called every time a message was received? Should the returned factory be
cached for reuse?
Stuart.
-> Process runtime increasing ->
26849 30911 39184 45096 52555 64386 java.util.HashMap$Entry
23261 32837 44761 28405 34285 29189 java.lang.String
10754 12122 15144 17748 20526 25229 java.util.HashMap
10745 12156 15194 17690 20460 25090 [Ljava.util.HashMap$Entry;
12327 14564 16157 16252 18389 22281 [I
7268 8042 10048 12098 14030 17384
org.apache.servicemix.jbi.messaging.InOutImpl
7268 8042 10048 12098 14030 17384
org.apache.servicemix.jbi.messaging.NormalizedMessageImpl
7272 8047 10054 12100 14032 17381
org.apache.servicemix.jbi.jaxp.StringSource
3636 4022 5024 6053 7019 8698
org.apache.servicemix.bean.support.Request
3634 4021 5024 6049 7015 8692
org.apache.servicemix.jbi.messaging.MessageExchangeFactoryImpl$PrettyCalenda
r
3634 4021 5024 6049 7015 8692
sun.util.calendar.Gregorian$Date
3634 4021 5024 6049 7015 8692
org.apache.servicemix.jbi.messaging.ExchangePacket
3634 4024 5026 6046 7011 8685 java.util.HashMap$EntrySet
3667 4098 5113 6030 6972 8578 [Z
3624 4067 5080 5975 6890 8439
java.util.concurrent.ConcurrentHashMap$HashEntry
3535 4005 5027 5875 6802 8352 java.util.HashSet
-----Original Message-----
From: Gert Vanthienen [mailto:gert.vanthienen@gmail.com]
Sent: 23 November 2009 09:30
To: users@servicemix.apache.org
Subject: Re: Not deallocating resources causing Old Gen memory leaks
L.S.,
I'm not seeing anything obvious missing from this snippet of code.
Have you tried taking a memory dump and use jmap or yourkit or something to
find the objects that are causing the memore leak?
Regards,
Gert Vanthienen
------------------------
Open Source SOA: http://fusesource.com
Blog: http://gertvanthienen.blogspot.com/
2009/11/20 Stuart Roe <st...@ovic.co.uk>:
> Hi all,
>
> I appear to have a memory leak while performing an exchange and I'm
> unsure how to identify it/correct it. Bascially, my application
> happily runs for a few days and then fails with an out of memory
> exception. Using jconsole I can see that the memory usage is bouncing
> around with a slow consistent creap up. This is why I suspect a memory
leak.
>
> The message path within the app is cxf-bc (ver. 2009.01) -> cxf-se (ver.
> 2009.01) all running in SMX 3.3.1/WinXP.
>
> The handler in CXF-SE creates a new InOut message and sync. posts it
> to the NMR, unmarshals the response into a reply. There is a copy of
> the handler function below. I have removed the actual data handling
> and
exceptions.
>
> My question is, I'm I missing some form of deallocation/close call
> that will return/release an object and hence remove my memory leak?
> I have found that replacing the message exchange from the handler and
> hard wiring a response fixes the leak.
>
> Any help/pointers would be helpful.
>
> Stuart.
>
> BTW:
> I'm using JiBX to marshal between NMR XML and Java objects (which may
> also be the source of the problem).
>
>
> public List<TypeConfigStatus> getConfigStatus()
> try {
> GetConfigStatusResponse response = null;
>
> DeliveryChannel ch = getContext().getDeliveryChannel();
> InOut exch = ch.createExchangeFactory().createInOutExchange();
> exch.setService(getService());
> exch.setInMessage(exch.createMessage());
> exch.getInMessage().setProperty("operation", "config-status");
> exch.getInMessage().setContent(new StringSource("<empty />"));
>
> if(ch.sendSync(exch)){
> NormalizedMessage amsg = exch.getOutMessage();
> try {
> SourceTransformer trans = new SourceTransformer();
> String content = trans.contentToString(amsg);
> IBindingFactory bfact =
> BindingDirectory.getFactory(GetConfigStatusResponse.class);
> IUnmarshallingContext uctx =
> bfact.createUnmarshallingContext();
> response = (GetConfigStatusResponse)
> uctx.unmarshalDocument(new StringReader(content));
>
> } catch
> // block removed
> }finally{
> exch.setStatus(ExchangeStatus.DONE);
> ch.send(exch);
> }
> }
>
> // set result = cooked(response)
>
> return result;
> }
>
>
RE: Not deallocating resources causing Old Gen memory leaks
Posted by Stuart Roe <st...@ovic.co.uk>.
Gert,
Thanks for the reply. I have taken your advise and used jmap's histo option
to collect information on the number of instances of objects. I ran a test
lastnight and have taken numerous histograms (jmap -histo) this morning and
have identified (I think) a memory leak in the handling of message exchange.
The histograms were created by hitting the GC button in jconsole and running
the jmap command straight afterwards (trying to reduce the noise from other
normal create->use->destroy operations). The following list shows the number
of instances for 6 samples followed by the name of the class. Unfortunately,
the period between the samples has not been consistent, so it's not possible
to calculte a leak rate.
There is a lot of noise on the java.lang.String class, but all the other
classes show a consistent increase in the number of instances and there is a
large number of servicemix/jbi/messaging related classes mentioned.
>From this, I conclude that there is:
A). a memory leak within the SMX 3.3.1/eip 2009.01/bean 2009.01.
B). my code is missing a close/release/dispose function call releasing the
resources.
Could I encounter a problem if the createExchangeFactory() function was
called every time a message was received? Should the returned factory be
cached for reuse?
Stuart.
-> Process runtime increasing ->
26849 30911 39184 45096 52555 64386 java.util.HashMap$Entry
23261 32837 44761 28405 34285 29189 java.lang.String
10754 12122 15144 17748 20526 25229 java.util.HashMap
10745 12156 15194 17690 20460 25090 [Ljava.util.HashMap$Entry;
12327 14564 16157 16252 18389 22281 [I
7268 8042 10048 12098 14030 17384
org.apache.servicemix.jbi.messaging.InOutImpl
7268 8042 10048 12098 14030 17384
org.apache.servicemix.jbi.messaging.NormalizedMessageImpl
7272 8047 10054 12100 14032 17381
org.apache.servicemix.jbi.jaxp.StringSource
3636 4022 5024 6053 7019 8698
org.apache.servicemix.bean.support.Request
3634 4021 5024 6049 7015 8692
org.apache.servicemix.jbi.messaging.MessageExchangeFactoryImpl$PrettyCalenda
r
3634 4021 5024 6049 7015 8692
sun.util.calendar.Gregorian$Date
3634 4021 5024 6049 7015 8692
org.apache.servicemix.jbi.messaging.ExchangePacket
3634 4024 5026 6046 7011 8685 java.util.HashMap$EntrySet
3667 4098 5113 6030 6972 8578 [Z
3624 4067 5080 5975 6890 8439
java.util.concurrent.ConcurrentHashMap$HashEntry
3535 4005 5027 5875 6802 8352 java.util.HashSet
-----Original Message-----
From: Gert Vanthienen [mailto:gert.vanthienen@gmail.com]
Sent: 23 November 2009 09:30
To: users@servicemix.apache.org
Subject: Re: Not deallocating resources causing Old Gen memory leaks
L.S.,
I'm not seeing anything obvious missing from this snippet of code.
Have you tried taking a memory dump and use jmap or yourkit or something to
find the objects that are causing the memore leak?
Regards,
Gert Vanthienen
------------------------
Open Source SOA: http://fusesource.com
Blog: http://gertvanthienen.blogspot.com/
2009/11/20 Stuart Roe <st...@ovic.co.uk>:
> Hi all,
>
> I appear to have a memory leak while performing an exchange and I'm
> unsure how to identify it/correct it. Bascially, my application
> happily runs for a few days and then fails with an out of memory
> exception. Using jconsole I can see that the memory usage is bouncing
> around with a slow consistent creap up. This is why I suspect a memory
leak.
>
> The message path within the app is cxf-bc (ver. 2009.01) -> cxf-se (ver.
> 2009.01) all running in SMX 3.3.1/WinXP.
>
> The handler in CXF-SE creates a new InOut message and sync. posts it
> to the NMR, unmarshals the response into a reply. There is a copy of
> the handler function below. I have removed the actual data handling and
exceptions.
>
> My question is, I'm I missing some form of deallocation/close call
> that will return/release an object and hence remove my memory leak?
> I have found that replacing the message exchange from the handler and
> hard wiring a response fixes the leak.
>
> Any help/pointers would be helpful.
>
> Stuart.
>
> BTW:
> I'm using JiBX to marshal between NMR XML and Java objects (which may
> also be the source of the problem).
>
>
> public List<TypeConfigStatus> getConfigStatus()
> try {
> GetConfigStatusResponse response = null;
>
> DeliveryChannel ch = getContext().getDeliveryChannel();
> InOut exch = ch.createExchangeFactory().createInOutExchange();
> exch.setService(getService());
> exch.setInMessage(exch.createMessage());
> exch.getInMessage().setProperty("operation", "config-status");
> exch.getInMessage().setContent(new StringSource("<empty />"));
>
> if(ch.sendSync(exch)){
> NormalizedMessage amsg = exch.getOutMessage();
> try {
> SourceTransformer trans = new SourceTransformer();
> String content = trans.contentToString(amsg);
> IBindingFactory bfact =
> BindingDirectory.getFactory(GetConfigStatusResponse.class);
> IUnmarshallingContext uctx =
> bfact.createUnmarshallingContext();
> response = (GetConfigStatusResponse)
> uctx.unmarshalDocument(new StringReader(content));
>
> } catch
> // block removed
> }finally{
> exch.setStatus(ExchangeStatus.DONE);
> ch.send(exch);
> }
> }
>
> // set result = cooked(response)
>
> return result;
> }
>
>
Re: Not deallocating resources causing Old Gen memory leaks
Posted by Gert Vanthienen <ge...@gmail.com>.
L.S.,
I'm not seeing anything obvious missing from this snippet of code.
Have you tried taking a memory dump and use jmap or yourkit or
something to find the objects that are causing the memore leak?
Regards,
Gert Vanthienen
------------------------
Open Source SOA: http://fusesource.com
Blog: http://gertvanthienen.blogspot.com/
2009/11/20 Stuart Roe <st...@ovic.co.uk>:
> Hi all,
>
> I appear to have a memory leak while performing an exchange and I'm unsure
> how to identify it/correct it. Bascially, my application happily runs for a
> few days and then fails with an out of memory exception. Using jconsole I
> can see that the memory usage is bouncing around with a slow consistent
> creap up. This is why I suspect a memory leak.
>
> The message path within the app is cxf-bc (ver. 2009.01) -> cxf-se (ver.
> 2009.01) all running in SMX 3.3.1/WinXP.
>
> The handler in CXF-SE creates a new InOut message and sync. posts it to the
> NMR, unmarshals the response into a reply. There is a copy of the handler
> function below. I have removed the actual data handling and exceptions.
>
> My question is, I'm I missing some form of deallocation/close call that will
> return/release an object and hence remove my memory leak?
> I have found that replacing the message exchange from the handler and hard
> wiring a response fixes the leak.
>
> Any help/pointers would be helpful.
>
> Stuart.
>
> BTW:
> I'm using JiBX to marshal between NMR XML and Java objects (which may also
> be the source of the problem).
>
>
> public List<TypeConfigStatus> getConfigStatus()
> try {
> GetConfigStatusResponse response = null;
>
> DeliveryChannel ch = getContext().getDeliveryChannel();
> InOut exch = ch.createExchangeFactory().createInOutExchange();
> exch.setService(getService());
> exch.setInMessage(exch.createMessage());
> exch.getInMessage().setProperty("operation", "config-status");
> exch.getInMessage().setContent(new StringSource("<empty />"));
>
> if(ch.sendSync(exch)){
> NormalizedMessage amsg = exch.getOutMessage();
> try {
> SourceTransformer trans = new SourceTransformer();
> String content = trans.contentToString(amsg);
> IBindingFactory bfact =
> BindingDirectory.getFactory(GetConfigStatusResponse.class);
> IUnmarshallingContext uctx = bfact.createUnmarshallingContext();
> response = (GetConfigStatusResponse) uctx.unmarshalDocument(new
> StringReader(content));
>
> } catch
> // block removed
> }finally{
> exch.setStatus(ExchangeStatus.DONE);
> ch.send(exch);
> }
> }
>
> // set result = cooked(response)
>
> return result;
> }
>
>