You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cxf.apache.org by Sergey Beryozkin <sb...@gmail.com> on 2014/07/04 18:43:21 UTC
Re: Classloader leak in ExtensionManagerBus (on tomee)
I've committed the fix to 2.6.14-SNAPSHOT
Cheers, Sergey
On 26/06/14 22:25, Sergey Beryozkin wrote:
> Hi,
> On 26/06/14 22:23, pablo.a.saavedra@gmail.com wrote:
>> Hi Sergey,
>>
>> sure, I'll apply the changes to our development Tomee server and see
>> how it
>> behaves. I'll probably won't get around to do that until tomorrow, so I
>> might have some results back next week.
>>
> Sounds good, we have a couple of weeks or so before 2.6.x gets released
>
> Thanks, Sergey
>> I'll keep you posted.
>> Thanks
>>
>>
>> On 26 June 2014 18:09, Sergey Beryozkin <sb...@gmail.com> wrote:
>>
>>> Hi
>>>
>>> On 26/06/14 19:12, pablo.a.saavedra@gmail.com wrote:
>>>
>>>> Thanks for following this up Sergey. I haven't dig much into the
>>>> code, but
>>>> from what I see, this is the part of the code where the proxy map is
>>>> being
>>>> created (AbstractResourceInfo#getProxyMap):
>>>>
>>>>
>>>> private <T> Map<Class<?>, Map<T, ThreadLocalProxy<?>>>
>>>>> getProxyMap(Class<T> keyCls, String prop, boolean create) {
>>>>> Object property = null;
>>>>> synchronized (bus) {
>>>>> property = bus.getProperty(prop);
>>>>> if (property == null && create) {
>>>>> Map<Class<?>, Map<T, ThreadLocalProxy<?>>> map
>>>>> = new ConcurrentHashMap<Class<?>, Map<T,
>>>>> ThreadLocalProxy<?>>>(2);
>>>>> bus.setProperty(prop, map);
>>>>> property = map;
>>>>> }
>>>>> }
>>>>> return (Map<Class<?>, Map<T, ThreadLocalProxy<?>>>)property;
>>>>> }
>>>>>
>>>>
>>>>
>>>> Do you really need a ConcurrentHashMap here? If not, it can be replaced
>>>> with a WeakHashMap and that should solve this particular issue. You can
>>>> also try and wrap a WeakHashMap with Collections.synchronizedMap,
>>>> but that
>>>> might impact performance if there's much contention.
>>>>
>>>> This was of good help, thanks. It might do a trick hopefully it will.
>>> We have ProviderFactory linking to providers stored on Endpoint, so
>>> getting the endpoint collected will ensure the custom provider
>>> classes will
>>> get unloaded.
>>>
>>> I've committed the update, For now I'll keep the change on the trunk
>>> only
>>> for a bit to ensure it has no side-effects and then push down to
>>> 2.7.x and
>>> 2.6.x before they get released.
>>>
>>>
>>> Let me know if I can be of any help.
>>>>
>>>
>>>
>>> If you could possibly rebuild 2.6.x (mvn install -Pfastinstall) with
>>> replacing all of ConcurrentHashMap with synhronized wrappers of
>>> WeakHashMap
>>> and see if it actually solves the issue in Tomee in the next few days or
>>> next week then it would help. The fix should actually work but it can
>>> make
>>> sense to double check...
>>>
>>> Thanks, Sergey
>>>
>>>
>>> Regards
>>>>
>>>>
>>>> On 26 June 2014 13:50, Sergey Beryozkin <sb...@gmail.com> wrote:
>>>>
>>>> On 26/06/14 17:28, Sergey Beryozkin wrote:
>>>>>
>>>>> Hi
>>>>>> On 26/06/14 13:11, pablo.a.saavedra@gmail.com wrote:
>>>>>>
>>>>>> Hi Sergey,
>>>>>>>
>>>>>>> thanks for your prompt response. You are right, what I saw in the
>>>>>>> heap
>>>>>>> dump
>>>>>>> were links to the provider class, which prevented the classloader
>>>>>>> from
>>>>>>> being garbage collected.
>>>>>>>
>>>>>>> The provider in question is registered via the standard JAX-RS
>>>>>>> means,
>>>>>>> in
>>>>>>> the getSingletons method of javax.ws.rs.core.Application. This is
>>>>>>> done
>>>>>>> this
>>>>>>> way because we need to configure the JacksonJaxbJsonProvider.
>>>>>>>
>>>>>>>
>>>>>>> I've confirmed that the context info JAX-RS provider stored on
>>>>>>> the Bus
>>>>>> contains the actual provider classes - this needs to be fixed.
>>>>>> I'll look into it
>>>>>>
>>>>>> This is ok in itself but due to Tomee having a default bus shared
>>>>> between
>>>>> the endpoints it becomes a problem after redeployments.
>>>>>
>>>>> Hmm... I'll need to think how to overcome it...
>>>>>
>>>>> Cheers, Sergey
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Thanks, Sergey
>>>>>>
>>>>>> I'll check with the Tomee team to see if endpoint specific
>>>>>> buses are
>>>>>>
>>>>>>> possible.
>>>>>>>
>>>>>>> Thanks a lot.
>>>>>>>
>>>>>>>
>>>>>>> On 26 June 2014 07:10, Sergey Beryozkin <sb...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> Hi
>>>>>>>
>>>>>>>>
>>>>>>>> On 26/06/14 04:32, pablo.a.saavedra@gmail.com wrote:
>>>>>>>>
>>>>>>>> Hi All,
>>>>>>>>
>>>>>>>>>
>>>>>>>>> hope you are doing well. We've been using Tomee as our application
>>>>>>>>> server
>>>>>>>>> lately (we have some basic JAX-RS APIs), and after several
>>>>>>>>> redeployments
>>>>>>>>> we
>>>>>>>>> get a PermGen space error. I've been digging into the heap
>>>>>>>>> dump, and
>>>>>>>>> from
>>>>>>>>> what I see our custom JAX-RS provider (JacksonJaxbJsonProvider) is
>>>>>>>>> being
>>>>>>>>> referenced inside ExtensionManagerBus's properties and never
>>>>>>>>> unregistered.
>>>>>>>>>
>>>>>>>>> I'm pretty sure it's Tomee's fault, because it should be doing due
>>>>>>>>> cleanup
>>>>>>>>> on undeployment, but I was wondering whether the
>>>>>>>>> ExtensionManagerBus
>>>>>>>>> has
>>>>>>>>> any way of removing the registered providers.
>>>>>>>>>
>>>>>>>>> CXF JAX-RS checks provider context properties when it
>>>>>>>>> registers
>>>>>>>>>
>>>>>>>>> providers and stores reflection-specific information and
>>>>>>>>> proxies on
>>>>>>>> the
>>>>>>>> bus, it does not store the actual providers.
>>>>>>>>
>>>>>>>> Do you have some more info how Bus ends up linking to the
>>>>>>>> provider ?
>>>>>>>> We have a feature allowing providers registered directly on the bus
>>>>>>>> via
>>>>>>>> bus properties, is it what is being done in your case ?
>>>>>>>>
>>>>>>>> If not then I'm not sure. ProviderFactory holding providers is
>>>>>>>> registered
>>>>>>>> on the endpoint but I'm not seeing the code where the endpoint is
>>>>>>>> registered on the bus.
>>>>>>>>
>>>>>>>> Either way, the problem appears to be originating from the fact
>>>>>>>> that a
>>>>>>>> single shared bus is used between multiple endpoints in Tomee.
>>>>>>>> Is it
>>>>>>>> possible in Tomee endpoint descriptors set up an endpoint specific
>>>>>>>> bus ?
>>>>>>>> For example, in Spring/Blueprint descriptors which can declare a
>>>>>>>> new
>>>>>>>> CXF
>>>>>>>> Bus and have jaxrs/jaxws endpoints linking to it and this bus
>>>>>>>> would be
>>>>>>>> recycled on the redeployment.
>>>>>>>>
>>>>>>>> So, please let me know:
>>>>>>>> - if you have more info about the link from Bus to providers
>>>>>>>> - check if providers are registered directly on the bus (check its
>>>>>>>> properties like "javax.ws.rs.ext.MessageBodyReader")
>>>>>>>> - check if Tomee allows creating endpoint specific buses
>>>>>>>>
>>>>>>>> Lets us know please how it goes
>>>>>>>>
>>>>>>>> Thanks, Sergey
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Any help would be appreaciated
>>>>>>>>
>>>>>>>> Thanks in advance.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>
>>>
>>> --
>>> Sergey Beryozkin
>>>
>>> Talend Community Coders
>>> http://coders.talend.com/
>>>
>>> Blog: http://sberyozkin.blogspot.com
>>>
>>
>
>
--
Sergey Beryozkin
Talend Community Coders
http://coders.talend.com/
Blog: http://sberyozkin.blogspot.com
Re: Classloader leak in ExtensionManagerBus (on tomee)
Posted by "pablo.a.saavedra@gmail.com" <pa...@gmail.com>.
Thank you Sergey, I'll take a look into it and let you know. Unfortunately
I wasn't able to test the fix during this week.
Regards
On 4 July 2014 13:43, Sergey Beryozkin <sb...@gmail.com> wrote:
> I've committed the fix to 2.6.14-SNAPSHOT
>
> Cheers, Sergey
>
> On 26/06/14 22:25, Sergey Beryozkin wrote:
>
>> Hi,
>> On 26/06/14 22:23, pablo.a.saavedra@gmail.com wrote:
>>
>>> Hi Sergey,
>>>
>>> sure, I'll apply the changes to our development Tomee server and see
>>> how it
>>> behaves. I'll probably won't get around to do that until tomorrow, so I
>>> might have some results back next week.
>>>
>>> Sounds good, we have a couple of weeks or so before 2.6.x gets released
>>
>> Thanks, Sergey
>>
>>> I'll keep you posted.
>>> Thanks
>>>
>>>
>>> On 26 June 2014 18:09, Sergey Beryozkin <sb...@gmail.com> wrote:
>>>
>>> Hi
>>>>
>>>> On 26/06/14 19:12, pablo.a.saavedra@gmail.com wrote:
>>>>
>>>> Thanks for following this up Sergey. I haven't dig much into the
>>>>> code, but
>>>>> from what I see, this is the part of the code where the proxy map is
>>>>> being
>>>>> created (AbstractResourceInfo#getProxyMap):
>>>>>
>>>>>
>>>>> private <T> Map<Class<?>, Map<T, ThreadLocalProxy<?>>>
>>>>>
>>>>>> getProxyMap(Class<T> keyCls, String prop, boolean create) {
>>>>>> Object property = null;
>>>>>> synchronized (bus) {
>>>>>> property = bus.getProperty(prop);
>>>>>> if (property == null && create) {
>>>>>> Map<Class<?>, Map<T, ThreadLocalProxy<?>>> map
>>>>>> = new ConcurrentHashMap<Class<?>, Map<T,
>>>>>> ThreadLocalProxy<?>>>(2);
>>>>>> bus.setProperty(prop, map);
>>>>>> property = map;
>>>>>> }
>>>>>> }
>>>>>> return (Map<Class<?>, Map<T, ThreadLocalProxy<?>>>)
>>>>>> property;
>>>>>> }
>>>>>>
>>>>>>
>>>>>
>>>>> Do you really need a ConcurrentHashMap here? If not, it can be replaced
>>>>> with a WeakHashMap and that should solve this particular issue. You can
>>>>> also try and wrap a WeakHashMap with Collections.synchronizedMap,
>>>>> but that
>>>>> might impact performance if there's much contention.
>>>>>
>>>>> This was of good help, thanks. It might do a trick hopefully it will.
>>>>>
>>>> We have ProviderFactory linking to providers stored on Endpoint, so
>>>> getting the endpoint collected will ensure the custom provider
>>>> classes will
>>>> get unloaded.
>>>>
>>>> I've committed the update, For now I'll keep the change on the trunk
>>>> only
>>>> for a bit to ensure it has no side-effects and then push down to
>>>> 2.7.x and
>>>> 2.6.x before they get released.
>>>>
>>>>
>>>> Let me know if I can be of any help.
>>>>
>>>>>
>>>>>
>>>>
>>>> If you could possibly rebuild 2.6.x (mvn install -Pfastinstall) with
>>>> replacing all of ConcurrentHashMap with synhronized wrappers of
>>>> WeakHashMap
>>>> and see if it actually solves the issue in Tomee in the next few days or
>>>> next week then it would help. The fix should actually work but it can
>>>> make
>>>> sense to double check...
>>>>
>>>> Thanks, Sergey
>>>>
>>>>
>>>> Regards
>>>>
>>>>>
>>>>>
>>>>> On 26 June 2014 13:50, Sergey Beryozkin <sb...@gmail.com> wrote:
>>>>>
>>>>> On 26/06/14 17:28, Sergey Beryozkin wrote:
>>>>>
>>>>>>
>>>>>> Hi
>>>>>>
>>>>>>> On 26/06/14 13:11, pablo.a.saavedra@gmail.com wrote:
>>>>>>>
>>>>>>> Hi Sergey,
>>>>>>>
>>>>>>>>
>>>>>>>> thanks for your prompt response. You are right, what I saw in the
>>>>>>>> heap
>>>>>>>> dump
>>>>>>>> were links to the provider class, which prevented the classloader
>>>>>>>> from
>>>>>>>> being garbage collected.
>>>>>>>>
>>>>>>>> The provider in question is registered via the standard JAX-RS
>>>>>>>> means,
>>>>>>>> in
>>>>>>>> the getSingletons method of javax.ws.rs.core.Application. This is
>>>>>>>> done
>>>>>>>> this
>>>>>>>> way because we need to configure the JacksonJaxbJsonProvider.
>>>>>>>>
>>>>>>>>
>>>>>>>> I've confirmed that the context info JAX-RS provider stored on
>>>>>>>> the Bus
>>>>>>>>
>>>>>>> contains the actual provider classes - this needs to be fixed.
>>>>>>> I'll look into it
>>>>>>>
>>>>>>> This is ok in itself but due to Tomee having a default bus shared
>>>>>>>
>>>>>> between
>>>>>> the endpoints it becomes a problem after redeployments.
>>>>>>
>>>>>> Hmm... I'll need to think how to overcome it...
>>>>>>
>>>>>> Cheers, Sergey
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Thanks, Sergey
>>>>>>
>>>>>>>
>>>>>>> I'll check with the Tomee team to see if endpoint specific
>>>>>>> buses are
>>>>>>>
>>>>>>> possible.
>>>>>>>>
>>>>>>>> Thanks a lot.
>>>>>>>>
>>>>>>>>
>>>>>>>> On 26 June 2014 07:10, Sergey Beryozkin <sb...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Hi
>>>>>>>>
>>>>>>>>
>>>>>>>>> On 26/06/14 04:32, pablo.a.saavedra@gmail.com wrote:
>>>>>>>>>
>>>>>>>>> Hi All,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> hope you are doing well. We've been using Tomee as our application
>>>>>>>>>> server
>>>>>>>>>> lately (we have some basic JAX-RS APIs), and after several
>>>>>>>>>> redeployments
>>>>>>>>>> we
>>>>>>>>>> get a PermGen space error. I've been digging into the heap
>>>>>>>>>> dump, and
>>>>>>>>>> from
>>>>>>>>>> what I see our custom JAX-RS provider (JacksonJaxbJsonProvider) is
>>>>>>>>>> being
>>>>>>>>>> referenced inside ExtensionManagerBus's properties and never
>>>>>>>>>> unregistered.
>>>>>>>>>>
>>>>>>>>>> I'm pretty sure it's Tomee's fault, because it should be doing due
>>>>>>>>>> cleanup
>>>>>>>>>> on undeployment, but I was wondering whether the
>>>>>>>>>> ExtensionManagerBus
>>>>>>>>>> has
>>>>>>>>>> any way of removing the registered providers.
>>>>>>>>>>
>>>>>>>>>> CXF JAX-RS checks provider context properties when it
>>>>>>>>>> registers
>>>>>>>>>>
>>>>>>>>>> providers and stores reflection-specific information and
>>>>>>>>>> proxies on
>>>>>>>>>>
>>>>>>>>> the
>>>>>>>>> bus, it does not store the actual providers.
>>>>>>>>>
>>>>>>>>> Do you have some more info how Bus ends up linking to the
>>>>>>>>> provider ?
>>>>>>>>> We have a feature allowing providers registered directly on the bus
>>>>>>>>> via
>>>>>>>>> bus properties, is it what is being done in your case ?
>>>>>>>>>
>>>>>>>>> If not then I'm not sure. ProviderFactory holding providers is
>>>>>>>>> registered
>>>>>>>>> on the endpoint but I'm not seeing the code where the endpoint is
>>>>>>>>> registered on the bus.
>>>>>>>>>
>>>>>>>>> Either way, the problem appears to be originating from the fact
>>>>>>>>> that a
>>>>>>>>> single shared bus is used between multiple endpoints in Tomee.
>>>>>>>>> Is it
>>>>>>>>> possible in Tomee endpoint descriptors set up an endpoint specific
>>>>>>>>> bus ?
>>>>>>>>> For example, in Spring/Blueprint descriptors which can declare a
>>>>>>>>> new
>>>>>>>>> CXF
>>>>>>>>> Bus and have jaxrs/jaxws endpoints linking to it and this bus
>>>>>>>>> would be
>>>>>>>>> recycled on the redeployment.
>>>>>>>>>
>>>>>>>>> So, please let me know:
>>>>>>>>> - if you have more info about the link from Bus to providers
>>>>>>>>> - check if providers are registered directly on the bus (check its
>>>>>>>>> properties like "javax.ws.rs.ext.MessageBodyReader")
>>>>>>>>> - check if Tomee allows creating endpoint specific buses
>>>>>>>>>
>>>>>>>>> Lets us know please how it goes
>>>>>>>>>
>>>>>>>>> Thanks, Sergey
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Any help would be appreaciated
>>>>>>>>>
>>>>>>>>> Thanks in advance.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>
>>>> --
>>>> Sergey Beryozkin
>>>>
>>>> Talend Community Coders
>>>> http://coders.talend.com/
>>>>
>>>> Blog: http://sberyozkin.blogspot.com
>>>>
>>>>
>>>
>>
>>
>
> --
> Sergey Beryozkin
>
> Talend Community Coders
> http://coders.talend.com/
>
> Blog: http://sberyozkin.blogspot.com
>