Re: [Axiom] Encountered OOM issue when service.xml has ServiceTCCL parameter.

[Geeth Munasinghe <ge...@gmail.com>]

Hi Andreas,

Thank you for the input. It was a mistake to send the same mail few times.
and I apologize for that.

We have osgified the axis2 and axiom, and we re not using the
wstx-asl-3.2.9.jar with in libs folder because we had some other issues
with that jar. Due to that reason in [1] method of [2] class,
XMLInputFactory.newInstance() returns default stax implementation
(com.sun.xml.internal.stream.XMLInputFactoryImpl). This default stax
implementations has a reference (not a direct one) to
MultiParentClassLoader. So this causes the OOM issue.

I have reproduced it with following steps.

   1. Put the axis2.war distribution to tomcat7 (Start the server once to
   let the axis2.war extract)
   2. Remove the wstx-asl-3.2.9.jar from tomcat/webapps/axis2/WEB-INF/lib
   3. Put the attached axis2 services
   4. Send about 5000 requests to client service.


Even I put the osgified the wstx-asl-3.2.9.jar (regardless other issues we
had), XMLInputFactory.newInstance() does not pick the [3] by reading the
META-INF/services (spi)

We embedded the wstx-asl-3.2.9.jar into axiom osgified bundle, but it seems
it even does not resolve the issue.

How can we make wstx-asl-3.2.9.jar only visible to axiom and let the stax
implementation be [3] witch is available in wstx-asl-3.2.9.jar ?


[1] private static XMLInputFactory newXMLInputFactory(final ClassLoader
classLoader,
            final StAXParserConfiguration configuration)

[2] org.apache.axiom.om.util.StAXUtils

[3] com.ctc.wstx.stax.WstxInputFactory

Thanks
Geeth

On Fri, Sep 19, 2014 at 3:14 AM, Andreas Veithen <an...@gmail.com>
wrote:

> On a side note: sending the same message to the same mailing list multiple
> times in a short time interval generally doesn't increase the likelihood of
> getting a response.
>
> Andreas
>
>
> On Thu Sep 18 2014 at 4:17:39 PM Geeth Munasinghe <ge...@gmail.com>
> wrote:
>
>> Hi
>>
>> If anyone knows the answer for this question, please help.
>>
>> Thanks
>> Geeth
>>
>> On Wed, Sep 17, 2014 at 9:55 PM, Geeth Munasinghe <ge...@gmail.com>
>> wrote:
>>
>>> Hi all,
>>>
>>> We are using axis2-1.6.1 version, it uses axiom 1.2.11 version, We
>>> recently encountered a OOM problem with two services which uses ServiceTCCL
>>> parameter. We are using this parameter because both those services use
>>> spring and hibernate with them. Scenario is one service is calling the
>>> other service. So one service acts as the client.
>>>
>>> We analyzed the heap dumps and found out that OOM issue was caused by
>>> org.apache.axiom.om.util.
>>> StAXUtils class. There are few weakhashmaps used in that class. Two of
>>> them are
>>>
>>>    1. inputFactoryPerCLMap
>>>    2. outputFactoryPerCLMap
>>>
>>> And those two weak hashmaps has other weak hashmaps inside of them.
>>> Those inner hashmaps cause the issue.
>>>
>>> We found out that when we use ServiceTCCL parameter  in the service.xml,
>>> in AbstractMessageReceiver.java  [1] of axis2 (line number 152 - 170)
>>> creates a new class loader object of the MultiParentClassLoader [2] for
>>> every request. So in StAXUtils class [3] in axiom, methods [4] and [5] uses
>>> inner  weak hashmaps of inputFactoryPerCLMap,  outputFactoryPerCLMap and
>>> fill them with the class loader as the key and XMLInputFactory /
>>> XMLOutputFactory as the value.
>>>
>>> Because every request gets a new class loader object of the
>>> MultiParentClassLoader, their hash values are different. So they are keep
>>> getting filled into those inner weak hash maps. Because the same key
>>> (classloder instance) gets inserted into both weak hashmaps, garbage
>>> collector does not remove them. So server goes OOM, When we analyze the
>>> heap dumps we found out that java.util.WeakHashMap fills over 80% of the
>>> memory when it goes OOM.
>>>
>>> I have made fix in StAXUtils in axiom as follows.
>>>
>>> Instead map.get(cl) I change it to map.get(cl.getClass().getName())
>>>
>>> Instead map.put(cl, factory) I changed it to
>>> map.put(cl.getClass().getName(), factory)
>>>
>>> I have attached the fix (svn diff) here with email. I am not sure I have
>>> done the correct fix for the issue. But I found that it solves my problem.
>>> Can some one please verify weather I have done the correct fix.
>>>
>>> Please consider that upgrading to new axis2 is not a solution for us at
>>> the moment.
>>>
>>> [1]
>>> http://svn.apache.org/repos/asf/axis/axis2/java/core/tags/v1.6.1/modules/kernel/src/org/apache/axis2/receivers/AbstractMessageReceiver.java
>>>
>>> [2] org.apache.axis2.classloader.MultiParentClassLoader
>>>
>>> [3] org.apache.axiom.om.util.StAXUtils
>>>
>>> [4] getXMLInputFactory_perClassLoader(StAXParserConfiguration
>>> configuration)
>>>
>>> [5] getXMLOutputFactory_perClassLoader(StAXWriterConfiguration
>>> configuration)
>>>
>>>
>>> Thanks in advance.
>>>
>>> Geeth
>>>
>>
>>