You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@cxf.apache.org by DTaylor <Da...@Merge.com> on 2013/10/25 15:05:03 UTC

Potential Memory Leak

Good morning all,

We are experiencing a rather severe memory leak issue in the use of the
DocumentImpl class (whether it be the org.apache.xerces.dom.DocumentImpl or
the com.sun.org.apache.xerces.internal.dom.DocumentImpl class) with regards
to the userData property.

During our (simplified) testing, I have observed that the number of
DocumentImpl objects grows from 9 instances after a single call to at least
48 after 50 calls of a service.  In addition, one particular DocumentImpl
object grows from 99 kb of retained heap to 156kb of retained heap after 50
calls, with a forced garbage collection not reducing the size at all,
growing at a rate of ~ 1.2kb of retained heap per call.  Upon further
inspection, the userData table contains numerous copies of WSDL Element
nodes (one brief inspection showed at least 4 copies of each claimType in
our simple test).  I'm not sure if this property grows as a percentage rate
of the WSDL size (our WSDL is 12K, 1.2k being 10%) making larger WSDLs leak
memory at a much more rapid rate or if the retained heap will grow at a
constant rate regardless of WSDL size but given the contents I would suggest
it is a function of WSDL size.

As an example of how serious this can become, our production service which
brought this to our attention had, at one point, over 500 MB of retained
heap in a single DocumentImpl userData property, with over 1 million objects
in the table.

Is this a known issue?  Is there a common configuration issue which may
cause this?

Thanks,

Dan.



--
View this message in context: http://cxf.547215.n5.nabble.com/Potential-Memory-Leak-tp5735592.html
Sent from the cxf-user mailing list archive at Nabble.com.

Re: Potential Memory Leak

Posted by DTaylor <Da...@Merge.com>.

Thanks Marcel but we are trying to narrow this down now as well.

Has anyone seen this behaviour before, however?

Dan



--
View this message in context: http://cxf.547215.n5.nabble.com/Potential-Memory-Leak-tp5735592p5735649.html
Sent from the cxf-user mailing list archive at Nabble.com.

Re: Potential Memory Leak

Posted by Marcel Stör <ma...@frightanic.com>.

Sorry I cannot comment on the problem itself (never experienced this).
However, if you're hunting (potential) memory leaks I recommend you give 
http://plumbr.eu/ a try.

I'm not associated with Plumbr in any way other than being a really 
happy customer.

On 25.10.2013 15:05, DTaylor wrote:
> Good morning all,
>
> We are experiencing a rather severe memory leak issue in the use of the
> DocumentImpl class (whether it be the org.apache.xerces.dom.DocumentImpl or
> the com.sun.org.apache.xerces.internal.dom.DocumentImpl class) with regards
> to the userData property.
>
> During our (simplified) testing, I have observed that the number of
> DocumentImpl objects grows from 9 instances after a single call to at least
> 48 after 50 calls of a service.  In addition, one particular DocumentImpl
> object grows from 99 kb of retained heap to 156kb of retained heap after 50
> calls, with a forced garbage collection not reducing the size at all,
> growing at a rate of ~ 1.2kb of retained heap per call.  Upon further
> inspection, the userData table contains numerous copies of WSDL Element
> nodes (one brief inspection showed at least 4 copies of each claimType in
> our simple test).  I'm not sure if this property grows as a percentage rate
> of the WSDL size (our WSDL is 12K, 1.2k being 10%) making larger WSDLs leak
> memory at a much more rapid rate or if the retained heap will grow at a
> constant rate regardless of WSDL size but given the contents I would suggest
> it is a function of WSDL size.
>
> As an example of how serious this can become, our production service which
> brought this to our attention had, at one point, over 500 MB of retained
> heap in a single DocumentImpl userData property, with over 1 million objects
> in the table.
>
> Is this a known issue?  Is there a common configuration issue which may
> cause this?
>
> Thanks,
>
> Dan.
>
>
>
> --
> View this message in context: http://cxf.547215.n5.nabble.com/Potential-Memory-Leak-tp5735592.html
> Sent from the cxf-user mailing list archive at Nabble.com.
>


-- 
Marcel Stör, http://frightanic.com
Couchsurfing: http://www.couchsurfing.com/people/marcelstoer
O< ascii ribbon campaign - stop html mail - www.asciiribbon.org

Re: Potential Memory Leak

Posted by Colm O hEigeartaigh <co...@apache.org>.

Hi Dan,

Are you explicitly plugging in the MemoryOnlyTokenStore? Any reason you are
not using the EhCache based version instead? What are your security +
caching requirements? See the following two blog entries on security token
caching in CXF:

http://coheigea.blogspot.ie/2012/04/security-token-caching-in-apache-cxf-26.html
http://coheigea.blogspot.ie/2012/04/security-token-caching-in-apache-cxf-26_25.html

Colm.




On Wed, Oct 30, 2013 at 12:04 PM, DTaylor <Da...@merge.com> wrote:

> Hi Daniel,
>
> We resolved the ever-growing DocumentImpl.  This issue turned out to be in
> one of our interceptors where we were using cloneNode(true) rather than
> importNode and the main DocumentImpl was acquiring copy after copy of the
> WSDL claimType nodes.
>
> We are still, however, experiencing the issue with many DocumentImpl
> objects
> being created (they turn out to be SecurityToken objects) and kept
> indefinitely.  For our purposes, we have need of multiple proxies, which I
> believe it is documented somewhere that this issue might exist in this
> scenario.
>
> Is there any documentation or thoughts on how to avoid having these extra
> SecurityToken objects remain in the MemoryOnlyTokenStore indefinitely (or
> at
> least seemingly so)?
>
> Thanks,
>
> Dan.
>
>
>
> --
> View this message in context:
> http://cxf.547215.n5.nabble.com/Potential-Memory-Leak-tp5735592p5735750.html
> Sent from the cxf-user mailing list archive at Nabble.com.
>



-- 
Colm O hEigeartaigh

Talend Community Coder
http://coders.talend.com

Re: Potential Memory Leak

Posted by Colm O hEigeartaigh <co...@gmail.com>.

Hi Dan,

This may have been fixed in a more recent version of CXF/WSS4J. If you are
concerned about the constant creation of new Crypto objects per-thread,
have you considered just creating it once + supplying the object as a
reference to CXF/WSS4J? For example, for the WS-SecurityPolicy approach you
can pass a Crypto Object in via the "ws-security.signature.crypto" tag for
Signature + "ws-security.encryption.crypto" tag for Encryption. Similar
tags exist for the older "Action" based approach of configuring WS-Security.

Colm.

On Mon, Nov 4, 2013 at 2:24 PM, DTaylor <Da...@merge.com> wrote:

> Hi Colm,
>
> Our user is creating a thread per invocation.  This new thread causes a new
> crypto object to be created.  This is with CXF 2.6.2, however, so the issue
> may have been fixed in a later release which we have not been able to move
> to as of yet.  Is this the case?
>
> Thanks,
>
> Dan.
>
>
>
>
> --
> View this message in context:
> http://cxf.547215.n5.nabble.com/Potential-Memory-Leak-tp5735592p5735913.html
> Sent from the cxf-user mailing list archive at Nabble.com.
>

Re: Potential Memory Leak

Posted by DTaylor <Da...@Merge.com>.

Hi Colm,

Our user is creating a thread per invocation.  This new thread causes a new
crypto object to be created.  This is with CXF 2.6.2, however, so the issue
may have been fixed in a later release which we have not been able to move
to as of yet.  Is this the case?

Thanks,

Dan.




--
View this message in context: http://cxf.547215.n5.nabble.com/Potential-Memory-Leak-tp5735592p5735913.html
Sent from the cxf-user mailing list archive at Nabble.com.

Re: Potential Memory Leak

Posted by Colm O hEigeartaigh <co...@apache.org>.

The Crypto objects should be cached in that manner to avoid loading them
per-invocation. Are you saying that it is repeatedly loading them + storing
them per-invocation?

Colm.


On Wed, Oct 30, 2013 at 6:44 PM, DTaylor <Da...@merge.com> wrote:

> Ok we've resolved this with the ehcache configuration...
>
> The one remaining leak we can see is that of Merlin crypto objects being
> placed into the WSHandler.cryptos hashtable.
>
> Has anyone encountered an issue where the Crypto objects are being indexed
> by the Crypto.toString() value (e.g. the key is:
> RefId-org.apache.ws.security.components.crypto.Merlin@<memory address>)?
>
> Thanks,
>
> Dan.
>
>
>
> --
> View this message in context:
> http://cxf.547215.n5.nabble.com/Potential-Memory-Leak-tp5735592p5735785.html
> Sent from the cxf-user mailing list archive at Nabble.com.
>



-- 
Colm O hEigeartaigh

Talend Community Coder
http://coders.talend.com

Re: Potential Memory Leak

Posted by DTaylor <Da...@Merge.com>.

Ok we've resolved this with the ehcache configuration...

The one remaining leak we can see is that of Merlin crypto objects being
placed into the WSHandler.cryptos hashtable.

Has anyone encountered an issue where the Crypto objects are being indexed
by the Crypto.toString() value (e.g. the key is: 
RefId-org.apache.ws.security.components.crypto.Merlin@<memory address>)?

Thanks,

Dan.



--
View this message in context: http://cxf.547215.n5.nabble.com/Potential-Memory-Leak-tp5735592p5735785.html
Sent from the cxf-user mailing list archive at Nabble.com.

Re: Potential Memory Leak

Posted by DTaylor <Da...@Merge.com>.

Hi Daniel,

We resolved the ever-growing DocumentImpl.  This issue turned out to be in
one of our interceptors where we were using cloneNode(true) rather than
importNode and the main DocumentImpl was acquiring copy after copy of the
WSDL claimType nodes.

We are still, however, experiencing the issue with many DocumentImpl objects
being created (they turn out to be SecurityToken objects) and kept
indefinitely.  For our purposes, we have need of multiple proxies, which I
believe it is documented somewhere that this issue might exist in this
scenario.

Is there any documentation or thoughts on how to avoid having these extra
SecurityToken objects remain in the MemoryOnlyTokenStore indefinitely (or at
least seemingly so)?

Thanks,

Dan.



--
View this message in context: http://cxf.547215.n5.nabble.com/Potential-Memory-Leak-tp5735592p5735750.html
Sent from the cxf-user mailing list archive at Nabble.com.

Re: Potential Memory Leak

Posted by Daniel Kulp <dk...@apache.org>.

Definitely not something I’ve seen before.  What’s really curious to me is why anything WSDL related would be “growing” on a per call basis on the server side.   The wsdl is setup and created at deploy time.   Nothing is cloned or anything per-call for that.   If you can create a sample that shows this problem, that would be great.

Dan



On Oct 25, 2013, at 9:05 AM, DTaylor <Da...@Merge.com> wrote:

> Good morning all,
> 
> We are experiencing a rather severe memory leak issue in the use of the
> DocumentImpl class (whether it be the org.apache.xerces.dom.DocumentImpl or
> the com.sun.org.apache.xerces.internal.dom.DocumentImpl class) with regards
> to the userData property.
> 
> During our (simplified) testing, I have observed that the number of
> DocumentImpl objects grows from 9 instances after a single call to at least
> 48 after 50 calls of a service.  In addition, one particular DocumentImpl
> object grows from 99 kb of retained heap to 156kb of retained heap after 50
> calls, with a forced garbage collection not reducing the size at all,
> growing at a rate of ~ 1.2kb of retained heap per call.  Upon further
> inspection, the userData table contains numerous copies of WSDL Element
> nodes (one brief inspection showed at least 4 copies of each claimType in
> our simple test).  I'm not sure if this property grows as a percentage rate
> of the WSDL size (our WSDL is 12K, 1.2k being 10%) making larger WSDLs leak
> memory at a much more rapid rate or if the retained heap will grow at a
> constant rate regardless of WSDL size but given the contents I would suggest
> it is a function of WSDL size.
> 
> As an example of how serious this can become, our production service which
> brought this to our attention had, at one point, over 500 MB of retained
> heap in a single DocumentImpl userData property, with over 1 million objects
> in the table.
> 
> Is this a known issue?  Is there a common configuration issue which may
> cause this?
> 
> Thanks,
> 
> Dan.
> 
> 
> 
> --
> View this message in context: http://cxf.547215.n5.nabble.com/Potential-Memory-Leak-tp5735592.html
> Sent from the cxf-user mailing list archive at Nabble.com.

-- 
Daniel Kulp
dkulp@apache.org - http://dankulp.com/blog
Talend Community Coder - http://coders.talend.com

Re: Potential Memory Leak

Posted by DTaylor <Da...@Merge.com>.

We are using cxf 2.6.4 and it is on the client side we see this on a call
into a service.




--
View this message in context: http://cxf.547215.n5.nabble.com/Potential-Memory-Leak-tp5735592p5735709.html
Sent from the cxf-user mailing list archive at Nabble.com.

Re: Potential Memory Leak

Posted by Jason Pell <ja...@pellcorp.com>.

Also what cxf version are you using?
On 26/10/2013 12:05 AM, "DTaylor" <Da...@merge.com> wrote:

> Good morning all,
>
> We are experiencing a rather severe memory leak issue in the use of the
> DocumentImpl class (whether it be the org.apache.xerces.dom.DocumentImpl or
> the com.sun.org.apache.xerces.internal.dom.DocumentImpl class) with regards
> to the userData property.
>
> During our (simplified) testing, I have observed that the number of
> DocumentImpl objects grows from 9 instances after a single call to at least
> 48 after 50 calls of a service.  In addition, one particular DocumentImpl
> object grows from 99 kb of retained heap to 156kb of retained heap after 50
> calls, with a forced garbage collection not reducing the size at all,
> growing at a rate of ~ 1.2kb of retained heap per call.  Upon further
> inspection, the userData table contains numerous copies of WSDL Element
> nodes (one brief inspection showed at least 4 copies of each claimType in
> our simple test).  I'm not sure if this property grows as a percentage rate
> of the WSDL size (our WSDL is 12K, 1.2k being 10%) making larger WSDLs leak
> memory at a much more rapid rate or if the retained heap will grow at a
> constant rate regardless of WSDL size but given the contents I would
> suggest
> it is a function of WSDL size.
>
> As an example of how serious this can become, our production service which
> brought this to our attention had, at one point, over 500 MB of retained
> heap in a single DocumentImpl userData property, with over 1 million
> objects
> in the table.
>
> Is this a known issue?  Is there a common configuration issue which may
> cause this?
>
> Thanks,
>
> Dan.
>
>
>
> --
> View this message in context:
> http://cxf.547215.n5.nabble.com/Potential-Memory-Leak-tp5735592.html
> Sent from the cxf-user mailing list archive at Nabble.com.
>