You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ignite.apache.org by Alexei Scherbakov <al...@gmail.com> on 2016/04/05 19:36:05 UTC

Cleaning internal thread locals

Hello.

Currently I'm working on the IGNITE-967
<https://issues.apache.org/jira/browse/IGNITE-967>.

In fact, it is well known, easily reproduceable, ClassLoader memory leak
problem in managed environments, such as servlet containers.
See http://wiki.apache.org/tomcat/MemoryLeakProtection to read more on that.
In short, the code using thread locals is responsible for cleaning them
after request processing
to prevent memory leaks related to different lifetime of application and
thread in pool.
Also blindly reusing thread from pool can be dangerous, because current
ThreadLocal value can be different from expected default value.

One posiible way of solving the problem is using post-processing Filter,
which is responsible for cleaning all thread locals on current thread.

Another approach would be sticking with container's specific solution to
this problem.
On example, tomcat provides the configuration property
renewThreadsWhenStoppingContext,
which forces it to renew threads involved in servicing requests for
destroyed context, thus removing leaks.

What do you think?


--

Best regards,
Alexei Scherbakov

Re: Cleaning internal thread locals

Posted by Alexei Scherbakov <al...@gmail.com>.
Val,

As Denis mentioned, it may be very tedious and error-prone job to fix all
thread locals,
and I'm not sure we should do it right now.
Leaks are only occuring in managed environments,so it would be wise to
provide solution to this specific problem.
Actually, I'm working on such solution and it is almost done. It involves
cleaning of thread locals after the request using a servlet filter.
But it may be not good for performance.

Denis,

There is no way to do that safely. Messages in logs will never disappear
until leaks will be fixed.

I see another option here. It is possible to use separate thread pool to
access cache in managed environments.
Which solves the problem.

2016-04-06 22:16 GMT+03:00 Valentin Kulichenko <
valentin.kulichenko@gmail.com>:

> Guys,
>
> Thread locals that are not cleaned up is a bug in our code, because this is
> a memory leak. This started with Tomcat only because it has automatic
> detection of such leaks and informs about them. Other application servers
> can easily lack this feature.
>
> Alexei, can you create a unit test first and list all the problematic
> places in the code? I believe there will be no generic solution, we will
> have to resolve all different cases individually.
>
> -Val
>
> On Wed, Apr 6, 2016 at 2:51 AM, Denis Magda <dm...@gridgain.com> wrote:
>
> > Cleaning of thread local variables like the ones below should be avoided
> > otherwise the performance will be impacted
> >
> > OptimizedObjectStreamRegistry$StreamHolder
> > BinaryMemoryAllocatorChunk
> >
> > I wouldn't clean these variable even from tomcat filter when a Thread
> from
> > Tomcat's pool finished processing a request because this Thread can be
> used
> > to interact with Ignite later.
> >
> > Simple full-text search shows that Ignite uses ThreadLocals in tons of
> the
> > places. So from my understanding it doesn't make sense to investigate
> every
> > variable and decide how to deal with this.
> >
> > Alexei, is there any way to get from Tomcat its internal threads on
> > context destroy and clean all our ThreadLocals? If to clean variables
> from
> > the filter then we have to do it wisely avoiding cleaning of performance
> > related ThreadLocals.
> > If we use renewThreadsWhenStoppingContext will these messages disappear?
> >
> > --
> > Denis
> >
> >
> > On 4/6/2016 12:30 PM, Semyon Boikov wrote:
> >
> >> As I understand in many palces thread locals can be used for performance
> >> reasons, and it is not good idea to always clear it.
> >>
> >> Initial problem in IGNITE-967 was about tomcat, since tomcat provides
> >> solution for this problem do we really need fix something in Ignite?
> >>
> >> Valentin, you created this ticket, what do you think?
> >>
> >> On Wed, Apr 6, 2016 at 11:59 AM, Alexei Scherbakov <
> >> alexey.scherbakoff@gmail.com> wrote:
> >>
> >> It is a very good idea to try to fix internal thread local cleaning in
> all
> >>> places.
> >>> On example, it can be easily done in class GridSpinReadWriteLock
> >>> Just replace code in line 172 with
> >>> readLockEntryCnt.remove();
> >>> and add line 388 with:
> >>> if (update == 0) readLockEntryCnt.remove();
> >>> ( needs more testing possibly )
> >>>
> >>> I've provided you with the list of offending places ( which may not be
> >>> completed )
> >>> But it is definitely a good place to start.
> >>> Who will be responsible for fixiing it ?
> >>>
> >>>
> >>> 2016-04-06 11:50 GMT+03:00 Denis Magda <dm...@gridgain.com>:
> >>>
> >>> As far as I understand Alexei, the issue here is that Ignite kernal
> >>>> doesn't have list of all the threads that accessed it during its
> >>>>
> >>> lifetime.
> >>>
> >>>> So the kernal won't be able to iterate over all the threads (like
> >>>> threads
> >>>> from Tomcat thread pool) and perform all the cleaning. This is the
> >>>> issue.
> >>>>
> >>>> --
> >>>> Denis
> >>>>
> >>>>
> >>>> On 4/6/2016 11:47 AM, Artem Shutak wrote:
> >>>>
> >>>> Alexei,
> >>>>>
> >>>>> I would follow the Valentin's suggestion in the issue description and
> >>>>> would
> >>>>> add to Ignite tests a check on thread locals after stopping all
> grids.
> >>>>>
> >>>> And
> >>>
> >>>> then we will be able to fix thread local cleaning in all places.
> >>>>>
> >>>>> Thanks,
> >>>>> -- Artem --
> >>>>>
> >>>>> On Wed, Apr 6, 2016 at 11:18 AM, Alexei Scherbakov <
> >>>>> alexey.scherbakoff@gmail.com> wrote:
> >>>>>
> >>>>> Valentin,
> >>>>>
> >>>>>> I did test on demo web app where HttpSession is backed by Ignite
> cache
> >>>>>> and
> >>>>>> collect some logs.
> >>>>>> There are many places in Ignite where thread locals are used.
> >>>>>> Then cache methods are accessed in threads from servlet container
> >>>>>> pool,
> >>>>>> we
> >>>>>> get leaks.
> >>>>>> I don't think it will be easy to add thread locals cleaning in all
> >>>>>>
> >>>>> these
> >>>
> >>>> places without major code rewrite, but, for sure, it would be ideal
> >>>>>> solution.
> >>>>>>
> >>>>>>
> >>>>>> <E0><EF><F0> 05, 2016 4:21:52 PM
> >>>>>> org.apache.catalina.loader.WebappClassLoaderBase
> >>>>>> checkThreadLocalMapForLeaks
> >>>>>> SEVERE: The web application [/app1] created a ThreadLocal with key
> of
> >>>>>> type
> >>>>>> [org.apache.ignite.internal.util.GridSpinReadWriteLock$1] (value
> >>>>>> [org.apache.ignite.internal.util.GridSpinReadWriteLock$1@5fecc35f])
> >>>>>>
> >>>>> and
> >>>
> >>>> a
> >>>>>> value of type [java.lang.Integer] (value [0]) but failed to remove
> it
> >>>>>> when
> >>>>>> the web application was stopped. Threads are going to be renewed
> over
> >>>>>> time
> >>>>>> to try and avoid a probable memory leak.
> >>>>>> <E0><EF><F0> 05, 2016 4:21:52 PM
> >>>>>> org.apache.catalina.loader.WebappClassLoaderBase
> >>>>>> checkThreadLocalMapForLeaks
> >>>>>> SEVERE: The web application [/app1] created a ThreadLocal with key
> of
> >>>>>> type
> >>>>>> [org.apache.ignite.internal.util.tostring.GridToStringBuilder$1]
> >>>>>> (value
> >>>>>>
> >>>>>>
> >>>
> [org.apache.ignite.internal.util.tostring.GridToStringBuilder$1@4ecdfc04
> >>>
> >>>> ])
> >>>>>> and a value of type [java.util.LinkedList] (value
> >>>>>>
> >>>>>>
> >>>>>>
> >>>
> [[org.apache.ignite.internal.util.tostring.GridToStringThreadLocal@684a2791
> >>>
> >>>> ]])
> >>>>>> but failed to remove it when the web application was stopped.
> Threads
> >>>>>>
> >>>>> are
> >>>
> >>>> going to be renewed over time to try and avoid a probable memory leak.
> >>>>>> <E0><EF><F0> 05, 2016 4:21:52 PM
> >>>>>> org.apache.catalina.loader.WebappClassLoaderBase
> >>>>>> checkThreadLocalMapForLeaks
> >>>>>> SEVERE: The web application [/app1] created a ThreadLocal with key
> of
> >>>>>> type
> >>>>>> [org.apache.ignite.internal.binary.BinaryThreadLocalContext$1]
> (value
> >>>>>>
> [org.apache.ignite.internal.binary.BinaryThreadLocalContext$1@4e9300e7
> >>>>>>
> >>>>> ])
> >>>
> >>>> and a value of type
> >>>>>> [org.apache.ignite.internal.binary.BinaryThreadLocalContext] (value
> >>>>>> [org.apache.ignite.internal.binary.BinaryThreadLocalContext@7934a88
> ])
> >>>>>> but
> >>>>>> failed to remove it when the web application was stopped. Threads
> are
> >>>>>> going
> >>>>>> to be renewed over time to try and avoid a probable memory leak.
> >>>>>> <E0><EF><F0> 05, 2016 4:21:52 PM
> >>>>>> org.apache.catalina.loader.WebappClassLoaderBase
> >>>>>> checkThreadLocalMapForLeaks
> >>>>>> SEVERE: The web application [/app1] created a ThreadLocal with key
> of
> >>>>>> type
> >>>>>> [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@56502b72])
> and
> >>>>>> a
> >>>>>> value of type
> >>>>>>
> [org.apache.ignite.internal.binary.streams.BinaryMemoryAllocatorChunk]
> >>>>>> (value
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>
> [org.apache.ignite.internal.binary.streams.BinaryMemoryAllocatorChunk@4eab9d2f
> >>>
> >>>> ])
> >>>>>> but failed to remove it when the web application was stopped.
> Threads
> >>>>>>
> >>>>> are
> >>>
> >>>> going to be renewed over time to try and avoid a probable memory leak.
> >>>>>> <E0><EF><F0> 05, 2016 4:21:52 PM
> >>>>>> org.apache.catalina.loader.WebappClassLoaderBase
> >>>>>> checkThreadLocalMapForLeaks
> >>>>>> SEVERE: The web application [/app1] created a ThreadLocal with key
> of
> >>>>>> type
> >>>>>> [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@eb53c9d])
> and a
> >>>>>> value
> >>>>>> of type
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>
> [org.apache.ignite.marshaller.optimized.OptimizedObjectStreamRegistry.StreamHolder]
> >>>
> >>>> (value
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>
> [org.apache.ignite.marshaller.optimized.OptimizedObjectStreamRegistry$StreamHolder@5e2c15f7
> >>>
> >>>> ])
> >>>>>> but failed to remove it when the web application was stopped.
> Threads
> >>>>>>
> >>>>> are
> >>>
> >>>> going to be renewed over time to try and avoid a probable memory leak.
> >>>>>> <E0><EF><F0> 05, 2016 4:21:52 PM
> >>>>>> org.apache.catalina.loader.WebappClassLoaderBase
> >>>>>> checkThreadLocalMapForLeaks
> >>>>>> SEVERE: The web application [/app1] created a ThreadLocal with key
> of
> >>>>>> type
> >>>>>> [org.apache.ignite.internal.util.GridSpinReadWriteLock$1] (value
> >>>>>> [org.apache.ignite.internal.util.GridSpinReadWriteLock$1@55b9da28])
> >>>>>>
> >>>>> and
> >>>
> >>>> a
> >>>>>> value of type [java.lang.Integer] (value [0]) but failed to remove
> it
> >>>>>> when
> >>>>>> the web application was stopped. Threads are going to be renewed
> over
> >>>>>> time
> >>>>>> to try and avoid a probable memory leak.
> >>>>>> <E0><EF><F0> 05, 2016 4:21:52 PM
> >>>>>> org.apache.catalina.loader.WebappClassLoaderBase
> >>>>>> checkThreadLocalMapForLeaks
> >>>>>> SEVERE: The web application [/app1] created a ThreadLocal with key
> of
> >>>>>> type
> >>>>>> [org.apache.ignite.internal.util.StripedCompositeReadWriteLock$1]
> >>>>>>
> >>>>> (value
> >>>
> [org.apache.ignite.internal.util.StripedCompositeReadWriteLock$1@1ac73431
> >>>
> >>>> ])
> >>>>>> and a value of type [java.lang.Integer] (value [9]) but failed to
> >>>>>>
> >>>>> remove
> >>>
> >>>> it
> >>>>>> when the web application was stopped. Threads are going to be
> renewed
> >>>>>> over
> >>>>>> time to try and avoid a probable memory leak.
> >>>>>> SEVERE: The web application [/app1] created a ThreadLocal with key
> of
> >>>>>> type
> >>>>>> [org.apache.ignite.internal.processors.cache.GridCacheAdapter$5]
> >>>>>> (value
> >>>>>>
> >>>>>>
> >>>
> [org.apache.ignite.internal.processors.cache.GridCacheAdapter$5@28a1bd99
> >>>
> >>>> ])
> >>>>>> and a value of type
> >>>>>>
> >>>>>>
> >>>>>>
> >>>
> [org.apache.ignite.internal.processors.cache.GridCacheAdapter.FutureHolder]
> >>>
> >>>> (value
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>
> [org.apache.ignite.internal.processors.cache.GridCacheAdapter$FutureHolder@6763d70c
> >>>
> >>>> ])
> >>>>>> but failed to remove it when the web application was stopped.
> Threads
> >>>>>>
> >>>>> are
> >>>
> >>>> going to be renewed over time to try and avoid a probable memory leak.
> >>>>>>
> >>>>>> As a solution, I propose to implement servlet filter as a part of
> >>>>>>
> >>>>> Ignite
> >>>
> >>>> web integration, which will be responsible for cleaning up all thread
> >>>>>> locals after request processing.
> >>>>>> Filter must be executed last in filter chain to work correctly.
> >>>>>> It is not necessary to use the filter in case of Tomcat, because it
> >>>>>> provides own solution to the problem requiring special configuration
> >>>>>> parameter for context
> >>>>>> User should decide which approach is better to him.
> >>>>>>
> >>>>>>
> >>>>>> 2016-04-06 1:45 GMT+03:00 Valentin Kulichenko <
> >>>>>> valentin.kulichenko@gmail.com
> >>>>>>
> >>>>>> :
> >>>>>>> Alexei,
> >>>>>>>
> >>>>>>> I would start with defining which thread locals cause this and why
> >>>>>>>
> >>>>>> they
> >>>
> >>>> are
> >>>>>>
> >>>>>> not properly cleaned on node shutdown. Ideally, this should never
> >>>>>>> happen.
> >>>>>>>
> >>>>>>> -Val
> >>>>>>>
> >>>>>>> On Tue, Apr 5, 2016 at 10:36 AM, Alexei Scherbakov <
> >>>>>>> alexey.scherbakoff@gmail.com> wrote:
> >>>>>>>
> >>>>>>> Hello.
> >>>>>>>
> >>>>>>>> Currently I'm working on the IGNITE-967
> >>>>>>>> <https://issues.apache.org/jira/browse/IGNITE-967>.
> >>>>>>>>
> >>>>>>>> In fact, it is well known, easily reproduceable, ClassLoader
> memory
> >>>>>>>>
> >>>>>>>> leak
> >>>>>>> problem in managed environments, such as servlet containers.
> >>>>>>>
> >>>>>>>> See http://wiki.apache.org/tomcat/MemoryLeakProtection to read
> more
> >>>>>>>>
> >>>>>>> on
> >>>
> >>>> that.
> >>>>>>>> In short, the code using thread locals is responsible for cleaning
> >>>>>>>>
> >>>>>>> them
> >>>
> >>>> after request processing
> >>>>>>>> to prevent memory leaks related to different lifetime of
> application
> >>>>>>>>
> >>>>>>>> and
> >>>>>>> thread in pool.
> >>>>>>>
> >>>>>>>> Also blindly reusing thread from pool can be dangerous, because
> >>>>>>>>
> >>>>>>> current
> >>>
> >>>> ThreadLocal value can be different from expected default value.
> >>>>>>>>
> >>>>>>>> One posiible way of solving the problem is using post-processing
> >>>>>>>>
> >>>>>>>> Filter,
> >>>>>>> which is responsible for cleaning all thread locals on current
> >>>>>>> thread.
> >>>>>>>
> >>>>>>>> Another approach would be sticking with container's specific
> >>>>>>>> solution
> >>>>>>>>
> >>>>>>>> to
> >>>>>>> this problem.
> >>>>>>>
> >>>>>>>> On example, tomcat provides the configuration property
> >>>>>>>> renewThreadsWhenStoppingContext,
> >>>>>>>> which forces it to renew threads involved in servicing requests
> for
> >>>>>>>> destroyed context, thus removing leaks.
> >>>>>>>>
> >>>>>>>> What do you think?
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> --
> >>>>>>>>
> >>>>>>>> Best regards,
> >>>>>>>> Alexei Scherbakov
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> --
> >>>>>>
> >>>>>> Best regards,
> >>>>>> Alexei Scherbakov
> >>>>>>
> >>>>>>
> >>>>>>
> >>> --
> >>>
> >>> Best regards,
> >>> Alexei Scherbakov
> >>>
> >>>
> >
>



-- 

Best regards,
Alexei Scherbakov

Re: Cleaning internal thread locals

Posted by Valentin Kulichenko <va...@gmail.com>.
Guys,

Thread locals that are not cleaned up is a bug in our code, because this is
a memory leak. This started with Tomcat only because it has automatic
detection of such leaks and informs about them. Other application servers
can easily lack this feature.

Alexei, can you create a unit test first and list all the problematic
places in the code? I believe there will be no generic solution, we will
have to resolve all different cases individually.

-Val

On Wed, Apr 6, 2016 at 2:51 AM, Denis Magda <dm...@gridgain.com> wrote:

> Cleaning of thread local variables like the ones below should be avoided
> otherwise the performance will be impacted
>
> OptimizedObjectStreamRegistry$StreamHolder
> BinaryMemoryAllocatorChunk
>
> I wouldn't clean these variable even from tomcat filter when a Thread from
> Tomcat's pool finished processing a request because this Thread can be used
> to interact with Ignite later.
>
> Simple full-text search shows that Ignite uses ThreadLocals in tons of the
> places. So from my understanding it doesn't make sense to investigate every
> variable and decide how to deal with this.
>
> Alexei, is there any way to get from Tomcat its internal threads on
> context destroy and clean all our ThreadLocals? If to clean variables from
> the filter then we have to do it wisely avoiding cleaning of performance
> related ThreadLocals.
> If we use renewThreadsWhenStoppingContext will these messages disappear?
>
> --
> Denis
>
>
> On 4/6/2016 12:30 PM, Semyon Boikov wrote:
>
>> As I understand in many palces thread locals can be used for performance
>> reasons, and it is not good idea to always clear it.
>>
>> Initial problem in IGNITE-967 was about tomcat, since tomcat provides
>> solution for this problem do we really need fix something in Ignite?
>>
>> Valentin, you created this ticket, what do you think?
>>
>> On Wed, Apr 6, 2016 at 11:59 AM, Alexei Scherbakov <
>> alexey.scherbakoff@gmail.com> wrote:
>>
>> It is a very good idea to try to fix internal thread local cleaning in all
>>> places.
>>> On example, it can be easily done in class GridSpinReadWriteLock
>>> Just replace code in line 172 with
>>> readLockEntryCnt.remove();
>>> and add line 388 with:
>>> if (update == 0) readLockEntryCnt.remove();
>>> ( needs more testing possibly )
>>>
>>> I've provided you with the list of offending places ( which may not be
>>> completed )
>>> But it is definitely a good place to start.
>>> Who will be responsible for fixiing it ?
>>>
>>>
>>> 2016-04-06 11:50 GMT+03:00 Denis Magda <dm...@gridgain.com>:
>>>
>>> As far as I understand Alexei, the issue here is that Ignite kernal
>>>> doesn't have list of all the threads that accessed it during its
>>>>
>>> lifetime.
>>>
>>>> So the kernal won't be able to iterate over all the threads (like
>>>> threads
>>>> from Tomcat thread pool) and perform all the cleaning. This is the
>>>> issue.
>>>>
>>>> --
>>>> Denis
>>>>
>>>>
>>>> On 4/6/2016 11:47 AM, Artem Shutak wrote:
>>>>
>>>> Alexei,
>>>>>
>>>>> I would follow the Valentin's suggestion in the issue description and
>>>>> would
>>>>> add to Ignite tests a check on thread locals after stopping all grids.
>>>>>
>>>> And
>>>
>>>> then we will be able to fix thread local cleaning in all places.
>>>>>
>>>>> Thanks,
>>>>> -- Artem --
>>>>>
>>>>> On Wed, Apr 6, 2016 at 11:18 AM, Alexei Scherbakov <
>>>>> alexey.scherbakoff@gmail.com> wrote:
>>>>>
>>>>> Valentin,
>>>>>
>>>>>> I did test on demo web app where HttpSession is backed by Ignite cache
>>>>>> and
>>>>>> collect some logs.
>>>>>> There are many places in Ignite where thread locals are used.
>>>>>> Then cache methods are accessed in threads from servlet container
>>>>>> pool,
>>>>>> we
>>>>>> get leaks.
>>>>>> I don't think it will be easy to add thread locals cleaning in all
>>>>>>
>>>>> these
>>>
>>>> places without major code rewrite, but, for sure, it would be ideal
>>>>>> solution.
>>>>>>
>>>>>>
>>>>>> <E0><EF><F0> 05, 2016 4:21:52 PM
>>>>>> org.apache.catalina.loader.WebappClassLoaderBase
>>>>>> checkThreadLocalMapForLeaks
>>>>>> SEVERE: The web application [/app1] created a ThreadLocal with key of
>>>>>> type
>>>>>> [org.apache.ignite.internal.util.GridSpinReadWriteLock$1] (value
>>>>>> [org.apache.ignite.internal.util.GridSpinReadWriteLock$1@5fecc35f])
>>>>>>
>>>>> and
>>>
>>>> a
>>>>>> value of type [java.lang.Integer] (value [0]) but failed to remove it
>>>>>> when
>>>>>> the web application was stopped. Threads are going to be renewed over
>>>>>> time
>>>>>> to try and avoid a probable memory leak.
>>>>>> <E0><EF><F0> 05, 2016 4:21:52 PM
>>>>>> org.apache.catalina.loader.WebappClassLoaderBase
>>>>>> checkThreadLocalMapForLeaks
>>>>>> SEVERE: The web application [/app1] created a ThreadLocal with key of
>>>>>> type
>>>>>> [org.apache.ignite.internal.util.tostring.GridToStringBuilder$1]
>>>>>> (value
>>>>>>
>>>>>>
>>> [org.apache.ignite.internal.util.tostring.GridToStringBuilder$1@4ecdfc04
>>>
>>>> ])
>>>>>> and a value of type [java.util.LinkedList] (value
>>>>>>
>>>>>>
>>>>>>
>>> [[org.apache.ignite.internal.util.tostring.GridToStringThreadLocal@684a2791
>>>
>>>> ]])
>>>>>> but failed to remove it when the web application was stopped. Threads
>>>>>>
>>>>> are
>>>
>>>> going to be renewed over time to try and avoid a probable memory leak.
>>>>>> <E0><EF><F0> 05, 2016 4:21:52 PM
>>>>>> org.apache.catalina.loader.WebappClassLoaderBase
>>>>>> checkThreadLocalMapForLeaks
>>>>>> SEVERE: The web application [/app1] created a ThreadLocal with key of
>>>>>> type
>>>>>> [org.apache.ignite.internal.binary.BinaryThreadLocalContext$1] (value
>>>>>> [org.apache.ignite.internal.binary.BinaryThreadLocalContext$1@4e9300e7
>>>>>>
>>>>> ])
>>>
>>>> and a value of type
>>>>>> [org.apache.ignite.internal.binary.BinaryThreadLocalContext] (value
>>>>>> [org.apache.ignite.internal.binary.BinaryThreadLocalContext@7934a88])
>>>>>> but
>>>>>> failed to remove it when the web application was stopped. Threads are
>>>>>> going
>>>>>> to be renewed over time to try and avoid a probable memory leak.
>>>>>> <E0><EF><F0> 05, 2016 4:21:52 PM
>>>>>> org.apache.catalina.loader.WebappClassLoaderBase
>>>>>> checkThreadLocalMapForLeaks
>>>>>> SEVERE: The web application [/app1] created a ThreadLocal with key of
>>>>>> type
>>>>>> [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@56502b72]) and
>>>>>> a
>>>>>> value of type
>>>>>> [org.apache.ignite.internal.binary.streams.BinaryMemoryAllocatorChunk]
>>>>>> (value
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>> [org.apache.ignite.internal.binary.streams.BinaryMemoryAllocatorChunk@4eab9d2f
>>>
>>>> ])
>>>>>> but failed to remove it when the web application was stopped. Threads
>>>>>>
>>>>> are
>>>
>>>> going to be renewed over time to try and avoid a probable memory leak.
>>>>>> <E0><EF><F0> 05, 2016 4:21:52 PM
>>>>>> org.apache.catalina.loader.WebappClassLoaderBase
>>>>>> checkThreadLocalMapForLeaks
>>>>>> SEVERE: The web application [/app1] created a ThreadLocal with key of
>>>>>> type
>>>>>> [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@eb53c9d]) and a
>>>>>> value
>>>>>> of type
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>> [org.apache.ignite.marshaller.optimized.OptimizedObjectStreamRegistry.StreamHolder]
>>>
>>>> (value
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>> [org.apache.ignite.marshaller.optimized.OptimizedObjectStreamRegistry$StreamHolder@5e2c15f7
>>>
>>>> ])
>>>>>> but failed to remove it when the web application was stopped. Threads
>>>>>>
>>>>> are
>>>
>>>> going to be renewed over time to try and avoid a probable memory leak.
>>>>>> <E0><EF><F0> 05, 2016 4:21:52 PM
>>>>>> org.apache.catalina.loader.WebappClassLoaderBase
>>>>>> checkThreadLocalMapForLeaks
>>>>>> SEVERE: The web application [/app1] created a ThreadLocal with key of
>>>>>> type
>>>>>> [org.apache.ignite.internal.util.GridSpinReadWriteLock$1] (value
>>>>>> [org.apache.ignite.internal.util.GridSpinReadWriteLock$1@55b9da28])
>>>>>>
>>>>> and
>>>
>>>> a
>>>>>> value of type [java.lang.Integer] (value [0]) but failed to remove it
>>>>>> when
>>>>>> the web application was stopped. Threads are going to be renewed over
>>>>>> time
>>>>>> to try and avoid a probable memory leak.
>>>>>> <E0><EF><F0> 05, 2016 4:21:52 PM
>>>>>> org.apache.catalina.loader.WebappClassLoaderBase
>>>>>> checkThreadLocalMapForLeaks
>>>>>> SEVERE: The web application [/app1] created a ThreadLocal with key of
>>>>>> type
>>>>>> [org.apache.ignite.internal.util.StripedCompositeReadWriteLock$1]
>>>>>>
>>>>> (value
>>> [org.apache.ignite.internal.util.StripedCompositeReadWriteLock$1@1ac73431
>>>
>>>> ])
>>>>>> and a value of type [java.lang.Integer] (value [9]) but failed to
>>>>>>
>>>>> remove
>>>
>>>> it
>>>>>> when the web application was stopped. Threads are going to be renewed
>>>>>> over
>>>>>> time to try and avoid a probable memory leak.
>>>>>> SEVERE: The web application [/app1] created a ThreadLocal with key of
>>>>>> type
>>>>>> [org.apache.ignite.internal.processors.cache.GridCacheAdapter$5]
>>>>>> (value
>>>>>>
>>>>>>
>>> [org.apache.ignite.internal.processors.cache.GridCacheAdapter$5@28a1bd99
>>>
>>>> ])
>>>>>> and a value of type
>>>>>>
>>>>>>
>>>>>>
>>> [org.apache.ignite.internal.processors.cache.GridCacheAdapter.FutureHolder]
>>>
>>>> (value
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>> [org.apache.ignite.internal.processors.cache.GridCacheAdapter$FutureHolder@6763d70c
>>>
>>>> ])
>>>>>> but failed to remove it when the web application was stopped. Threads
>>>>>>
>>>>> are
>>>
>>>> going to be renewed over time to try and avoid a probable memory leak.
>>>>>>
>>>>>> As a solution, I propose to implement servlet filter as a part of
>>>>>>
>>>>> Ignite
>>>
>>>> web integration, which will be responsible for cleaning up all thread
>>>>>> locals after request processing.
>>>>>> Filter must be executed last in filter chain to work correctly.
>>>>>> It is not necessary to use the filter in case of Tomcat, because it
>>>>>> provides own solution to the problem requiring special configuration
>>>>>> parameter for context
>>>>>> User should decide which approach is better to him.
>>>>>>
>>>>>>
>>>>>> 2016-04-06 1:45 GMT+03:00 Valentin Kulichenko <
>>>>>> valentin.kulichenko@gmail.com
>>>>>>
>>>>>> :
>>>>>>> Alexei,
>>>>>>>
>>>>>>> I would start with defining which thread locals cause this and why
>>>>>>>
>>>>>> they
>>>
>>>> are
>>>>>>
>>>>>> not properly cleaned on node shutdown. Ideally, this should never
>>>>>>> happen.
>>>>>>>
>>>>>>> -Val
>>>>>>>
>>>>>>> On Tue, Apr 5, 2016 at 10:36 AM, Alexei Scherbakov <
>>>>>>> alexey.scherbakoff@gmail.com> wrote:
>>>>>>>
>>>>>>> Hello.
>>>>>>>
>>>>>>>> Currently I'm working on the IGNITE-967
>>>>>>>> <https://issues.apache.org/jira/browse/IGNITE-967>.
>>>>>>>>
>>>>>>>> In fact, it is well known, easily reproduceable, ClassLoader memory
>>>>>>>>
>>>>>>>> leak
>>>>>>> problem in managed environments, such as servlet containers.
>>>>>>>
>>>>>>>> See http://wiki.apache.org/tomcat/MemoryLeakProtection to read more
>>>>>>>>
>>>>>>> on
>>>
>>>> that.
>>>>>>>> In short, the code using thread locals is responsible for cleaning
>>>>>>>>
>>>>>>> them
>>>
>>>> after request processing
>>>>>>>> to prevent memory leaks related to different lifetime of application
>>>>>>>>
>>>>>>>> and
>>>>>>> thread in pool.
>>>>>>>
>>>>>>>> Also blindly reusing thread from pool can be dangerous, because
>>>>>>>>
>>>>>>> current
>>>
>>>> ThreadLocal value can be different from expected default value.
>>>>>>>>
>>>>>>>> One posiible way of solving the problem is using post-processing
>>>>>>>>
>>>>>>>> Filter,
>>>>>>> which is responsible for cleaning all thread locals on current
>>>>>>> thread.
>>>>>>>
>>>>>>>> Another approach would be sticking with container's specific
>>>>>>>> solution
>>>>>>>>
>>>>>>>> to
>>>>>>> this problem.
>>>>>>>
>>>>>>>> On example, tomcat provides the configuration property
>>>>>>>> renewThreadsWhenStoppingContext,
>>>>>>>> which forces it to renew threads involved in servicing requests for
>>>>>>>> destroyed context, thus removing leaks.
>>>>>>>>
>>>>>>>> What do you think?
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>>
>>>>>>>> Best regards,
>>>>>>>> Alexei Scherbakov
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>
>>>>>> Best regards,
>>>>>> Alexei Scherbakov
>>>>>>
>>>>>>
>>>>>>
>>> --
>>>
>>> Best regards,
>>> Alexei Scherbakov
>>>
>>>
>

Re: Cleaning internal thread locals

Posted by Denis Magda <dm...@gridgain.com>.
Cleaning of thread local variables like the ones below should be avoided 
otherwise the performance will be impacted

OptimizedObjectStreamRegistry$StreamHolder
BinaryMemoryAllocatorChunk

I wouldn't clean these variable even from tomcat filter when a Thread 
from Tomcat's pool finished processing a request because this Thread can 
be used to interact with Ignite later.

Simple full-text search shows that Ignite uses ThreadLocals in tons of 
the places. So from my understanding it doesn't make sense to 
investigate every variable and decide how to deal with this.

Alexei, is there any way to get from Tomcat its internal threads on 
context destroy and clean all our ThreadLocals? If to clean variables 
from the filter then we have to do it wisely avoiding cleaning of 
performance related ThreadLocals.
If we use renewThreadsWhenStoppingContext will these messages disappear?

--
Denis

On 4/6/2016 12:30 PM, Semyon Boikov wrote:
> As I understand in many palces thread locals can be used for performance
> reasons, and it is not good idea to always clear it.
>
> Initial problem in IGNITE-967 was about tomcat, since tomcat provides
> solution for this problem do we really need fix something in Ignite?
>
> Valentin, you created this ticket, what do you think?
>
> On Wed, Apr 6, 2016 at 11:59 AM, Alexei Scherbakov <
> alexey.scherbakoff@gmail.com> wrote:
>
>> It is a very good idea to try to fix internal thread local cleaning in all
>> places.
>> On example, it can be easily done in class GridSpinReadWriteLock
>> Just replace code in line 172 with
>> readLockEntryCnt.remove();
>> and add line 388 with:
>> if (update == 0) readLockEntryCnt.remove();
>> ( needs more testing possibly )
>>
>> I've provided you with the list of offending places ( which may not be
>> completed )
>> But it is definitely a good place to start.
>> Who will be responsible for fixiing it ?
>>
>>
>> 2016-04-06 11:50 GMT+03:00 Denis Magda <dm...@gridgain.com>:
>>
>>> As far as I understand Alexei, the issue here is that Ignite kernal
>>> doesn't have list of all the threads that accessed it during its
>> lifetime.
>>> So the kernal won't be able to iterate over all the threads (like threads
>>> from Tomcat thread pool) and perform all the cleaning. This is the issue.
>>>
>>> --
>>> Denis
>>>
>>>
>>> On 4/6/2016 11:47 AM, Artem Shutak wrote:
>>>
>>>> Alexei,
>>>>
>>>> I would follow the Valentin's suggestion in the issue description and
>>>> would
>>>> add to Ignite tests a check on thread locals after stopping all grids.
>> And
>>>> then we will be able to fix thread local cleaning in all places.
>>>>
>>>> Thanks,
>>>> -- Artem --
>>>>
>>>> On Wed, Apr 6, 2016 at 11:18 AM, Alexei Scherbakov <
>>>> alexey.scherbakoff@gmail.com> wrote:
>>>>
>>>> Valentin,
>>>>> I did test on demo web app where HttpSession is backed by Ignite cache
>>>>> and
>>>>> collect some logs.
>>>>> There are many places in Ignite where thread locals are used.
>>>>> Then cache methods are accessed in threads from servlet container pool,
>>>>> we
>>>>> get leaks.
>>>>> I don't think it will be easy to add thread locals cleaning in all
>> these
>>>>> places without major code rewrite, but, for sure, it would be ideal
>>>>> solution.
>>>>>
>>>>>
>>>>> <E0><EF><F0> 05, 2016 4:21:52 PM
>>>>> org.apache.catalina.loader.WebappClassLoaderBase
>>>>> checkThreadLocalMapForLeaks
>>>>> SEVERE: The web application [/app1] created a ThreadLocal with key of
>>>>> type
>>>>> [org.apache.ignite.internal.util.GridSpinReadWriteLock$1] (value
>>>>> [org.apache.ignite.internal.util.GridSpinReadWriteLock$1@5fecc35f])
>> and
>>>>> a
>>>>> value of type [java.lang.Integer] (value [0]) but failed to remove it
>>>>> when
>>>>> the web application was stopped. Threads are going to be renewed over
>>>>> time
>>>>> to try and avoid a probable memory leak.
>>>>> <E0><EF><F0> 05, 2016 4:21:52 PM
>>>>> org.apache.catalina.loader.WebappClassLoaderBase
>>>>> checkThreadLocalMapForLeaks
>>>>> SEVERE: The web application [/app1] created a ThreadLocal with key of
>>>>> type
>>>>> [org.apache.ignite.internal.util.tostring.GridToStringBuilder$1] (value
>>>>>
>> [org.apache.ignite.internal.util.tostring.GridToStringBuilder$1@4ecdfc04
>>>>> ])
>>>>> and a value of type [java.util.LinkedList] (value
>>>>>
>>>>>
>> [[org.apache.ignite.internal.util.tostring.GridToStringThreadLocal@684a2791
>>>>> ]])
>>>>> but failed to remove it when the web application was stopped. Threads
>> are
>>>>> going to be renewed over time to try and avoid a probable memory leak.
>>>>> <E0><EF><F0> 05, 2016 4:21:52 PM
>>>>> org.apache.catalina.loader.WebappClassLoaderBase
>>>>> checkThreadLocalMapForLeaks
>>>>> SEVERE: The web application [/app1] created a ThreadLocal with key of
>>>>> type
>>>>> [org.apache.ignite.internal.binary.BinaryThreadLocalContext$1] (value
>>>>> [org.apache.ignite.internal.binary.BinaryThreadLocalContext$1@4e9300e7
>> ])
>>>>> and a value of type
>>>>> [org.apache.ignite.internal.binary.BinaryThreadLocalContext] (value
>>>>> [org.apache.ignite.internal.binary.BinaryThreadLocalContext@7934a88])
>>>>> but
>>>>> failed to remove it when the web application was stopped. Threads are
>>>>> going
>>>>> to be renewed over time to try and avoid a probable memory leak.
>>>>> <E0><EF><F0> 05, 2016 4:21:52 PM
>>>>> org.apache.catalina.loader.WebappClassLoaderBase
>>>>> checkThreadLocalMapForLeaks
>>>>> SEVERE: The web application [/app1] created a ThreadLocal with key of
>>>>> type
>>>>> [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@56502b72]) and a
>>>>> value of type
>>>>> [org.apache.ignite.internal.binary.streams.BinaryMemoryAllocatorChunk]
>>>>> (value
>>>>>
>>>>>
>>>>>
>> [org.apache.ignite.internal.binary.streams.BinaryMemoryAllocatorChunk@4eab9d2f
>>>>> ])
>>>>> but failed to remove it when the web application was stopped. Threads
>> are
>>>>> going to be renewed over time to try and avoid a probable memory leak.
>>>>> <E0><EF><F0> 05, 2016 4:21:52 PM
>>>>> org.apache.catalina.loader.WebappClassLoaderBase
>>>>> checkThreadLocalMapForLeaks
>>>>> SEVERE: The web application [/app1] created a ThreadLocal with key of
>>>>> type
>>>>> [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@eb53c9d]) and a
>>>>> value
>>>>> of type
>>>>>
>>>>>
>>>>>
>> [org.apache.ignite.marshaller.optimized.OptimizedObjectStreamRegistry.StreamHolder]
>>>>> (value
>>>>>
>>>>>
>>>>>
>> [org.apache.ignite.marshaller.optimized.OptimizedObjectStreamRegistry$StreamHolder@5e2c15f7
>>>>> ])
>>>>> but failed to remove it when the web application was stopped. Threads
>> are
>>>>> going to be renewed over time to try and avoid a probable memory leak.
>>>>> <E0><EF><F0> 05, 2016 4:21:52 PM
>>>>> org.apache.catalina.loader.WebappClassLoaderBase
>>>>> checkThreadLocalMapForLeaks
>>>>> SEVERE: The web application [/app1] created a ThreadLocal with key of
>>>>> type
>>>>> [org.apache.ignite.internal.util.GridSpinReadWriteLock$1] (value
>>>>> [org.apache.ignite.internal.util.GridSpinReadWriteLock$1@55b9da28])
>> and
>>>>> a
>>>>> value of type [java.lang.Integer] (value [0]) but failed to remove it
>>>>> when
>>>>> the web application was stopped. Threads are going to be renewed over
>>>>> time
>>>>> to try and avoid a probable memory leak.
>>>>> <E0><EF><F0> 05, 2016 4:21:52 PM
>>>>> org.apache.catalina.loader.WebappClassLoaderBase
>>>>> checkThreadLocalMapForLeaks
>>>>> SEVERE: The web application [/app1] created a ThreadLocal with key of
>>>>> type
>>>>> [org.apache.ignite.internal.util.StripedCompositeReadWriteLock$1]
>> (value
>> [org.apache.ignite.internal.util.StripedCompositeReadWriteLock$1@1ac73431
>>>>> ])
>>>>> and a value of type [java.lang.Integer] (value [9]) but failed to
>> remove
>>>>> it
>>>>> when the web application was stopped. Threads are going to be renewed
>>>>> over
>>>>> time to try and avoid a probable memory leak.
>>>>> SEVERE: The web application [/app1] created a ThreadLocal with key of
>>>>> type
>>>>> [org.apache.ignite.internal.processors.cache.GridCacheAdapter$5] (value
>>>>>
>> [org.apache.ignite.internal.processors.cache.GridCacheAdapter$5@28a1bd99
>>>>> ])
>>>>> and a value of type
>>>>>
>>>>>
>> [org.apache.ignite.internal.processors.cache.GridCacheAdapter.FutureHolder]
>>>>> (value
>>>>>
>>>>>
>>>>>
>> [org.apache.ignite.internal.processors.cache.GridCacheAdapter$FutureHolder@6763d70c
>>>>> ])
>>>>> but failed to remove it when the web application was stopped. Threads
>> are
>>>>> going to be renewed over time to try and avoid a probable memory leak.
>>>>>
>>>>> As a solution, I propose to implement servlet filter as a part of
>> Ignite
>>>>> web integration, which will be responsible for cleaning up all thread
>>>>> locals after request processing.
>>>>> Filter must be executed last in filter chain to work correctly.
>>>>> It is not necessary to use the filter in case of Tomcat, because it
>>>>> provides own solution to the problem requiring special configuration
>>>>> parameter for context
>>>>> User should decide which approach is better to him.
>>>>>
>>>>>
>>>>> 2016-04-06 1:45 GMT+03:00 Valentin Kulichenko <
>>>>> valentin.kulichenko@gmail.com
>>>>>
>>>>>> :
>>>>>> Alexei,
>>>>>>
>>>>>> I would start with defining which thread locals cause this and why
>> they
>>>>> are
>>>>>
>>>>>> not properly cleaned on node shutdown. Ideally, this should never
>>>>>> happen.
>>>>>>
>>>>>> -Val
>>>>>>
>>>>>> On Tue, Apr 5, 2016 at 10:36 AM, Alexei Scherbakov <
>>>>>> alexey.scherbakoff@gmail.com> wrote:
>>>>>>
>>>>>> Hello.
>>>>>>> Currently I'm working on the IGNITE-967
>>>>>>> <https://issues.apache.org/jira/browse/IGNITE-967>.
>>>>>>>
>>>>>>> In fact, it is well known, easily reproduceable, ClassLoader memory
>>>>>>>
>>>>>> leak
>>>>>> problem in managed environments, such as servlet containers.
>>>>>>> See http://wiki.apache.org/tomcat/MemoryLeakProtection to read more
>> on
>>>>>>> that.
>>>>>>> In short, the code using thread locals is responsible for cleaning
>> them
>>>>>>> after request processing
>>>>>>> to prevent memory leaks related to different lifetime of application
>>>>>>>
>>>>>> and
>>>>>> thread in pool.
>>>>>>> Also blindly reusing thread from pool can be dangerous, because
>> current
>>>>>>> ThreadLocal value can be different from expected default value.
>>>>>>>
>>>>>>> One posiible way of solving the problem is using post-processing
>>>>>>>
>>>>>> Filter,
>>>>>> which is responsible for cleaning all thread locals on current thread.
>>>>>>> Another approach would be sticking with container's specific solution
>>>>>>>
>>>>>> to
>>>>>> this problem.
>>>>>>> On example, tomcat provides the configuration property
>>>>>>> renewThreadsWhenStoppingContext,
>>>>>>> which forces it to renew threads involved in servicing requests for
>>>>>>> destroyed context, thus removing leaks.
>>>>>>>
>>>>>>> What do you think?
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> Best regards,
>>>>>>> Alexei Scherbakov
>>>>>>>
>>>>>>>
>>>>> --
>>>>>
>>>>> Best regards,
>>>>> Alexei Scherbakov
>>>>>
>>>>>
>>
>> --
>>
>> Best regards,
>> Alexei Scherbakov
>>

Re: Cleaning internal thread locals

Posted by Semyon Boikov <sb...@gridgain.com>.
As I understand in many palces thread locals can be used for performance
reasons, and it is not good idea to always clear it.

Initial problem in IGNITE-967 was about tomcat, since tomcat provides
solution for this problem do we really need fix something in Ignite?

Valentin, you created this ticket, what do you think?

On Wed, Apr 6, 2016 at 11:59 AM, Alexei Scherbakov <
alexey.scherbakoff@gmail.com> wrote:

> It is a very good idea to try to fix internal thread local cleaning in all
> places.
> On example, it can be easily done in class GridSpinReadWriteLock
> Just replace code in line 172 with
> readLockEntryCnt.remove();
> and add line 388 with:
> if (update == 0) readLockEntryCnt.remove();
> ( needs more testing possibly )
>
> I've provided you with the list of offending places ( which may not be
> completed )
> But it is definitely a good place to start.
> Who will be responsible for fixiing it ?
>
>
> 2016-04-06 11:50 GMT+03:00 Denis Magda <dm...@gridgain.com>:
>
> > As far as I understand Alexei, the issue here is that Ignite kernal
> > doesn't have list of all the threads that accessed it during its
> lifetime.
> > So the kernal won't be able to iterate over all the threads (like threads
> > from Tomcat thread pool) and perform all the cleaning. This is the issue.
> >
> > --
> > Denis
> >
> >
> > On 4/6/2016 11:47 AM, Artem Shutak wrote:
> >
> >> Alexei,
> >>
> >> I would follow the Valentin's suggestion in the issue description and
> >> would
> >> add to Ignite tests a check on thread locals after stopping all grids.
> And
> >> then we will be able to fix thread local cleaning in all places.
> >>
> >> Thanks,
> >> -- Artem --
> >>
> >> On Wed, Apr 6, 2016 at 11:18 AM, Alexei Scherbakov <
> >> alexey.scherbakoff@gmail.com> wrote:
> >>
> >> Valentin,
> >>>
> >>> I did test on demo web app where HttpSession is backed by Ignite cache
> >>> and
> >>> collect some logs.
> >>> There are many places in Ignite where thread locals are used.
> >>> Then cache methods are accessed in threads from servlet container pool,
> >>> we
> >>> get leaks.
> >>> I don't think it will be easy to add thread locals cleaning in all
> these
> >>> places without major code rewrite, but, for sure, it would be ideal
> >>> solution.
> >>>
> >>>
> >>> <E0><EF><F0> 05, 2016 4:21:52 PM
> >>> org.apache.catalina.loader.WebappClassLoaderBase
> >>> checkThreadLocalMapForLeaks
> >>> SEVERE: The web application [/app1] created a ThreadLocal with key of
> >>> type
> >>> [org.apache.ignite.internal.util.GridSpinReadWriteLock$1] (value
> >>> [org.apache.ignite.internal.util.GridSpinReadWriteLock$1@5fecc35f])
> and
> >>> a
> >>> value of type [java.lang.Integer] (value [0]) but failed to remove it
> >>> when
> >>> the web application was stopped. Threads are going to be renewed over
> >>> time
> >>> to try and avoid a probable memory leak.
> >>> <E0><EF><F0> 05, 2016 4:21:52 PM
> >>> org.apache.catalina.loader.WebappClassLoaderBase
> >>> checkThreadLocalMapForLeaks
> >>> SEVERE: The web application [/app1] created a ThreadLocal with key of
> >>> type
> >>> [org.apache.ignite.internal.util.tostring.GridToStringBuilder$1] (value
> >>>
> [org.apache.ignite.internal.util.tostring.GridToStringBuilder$1@4ecdfc04
> >>> ])
> >>> and a value of type [java.util.LinkedList] (value
> >>>
> >>>
> [[org.apache.ignite.internal.util.tostring.GridToStringThreadLocal@684a2791
> >>> ]])
> >>> but failed to remove it when the web application was stopped. Threads
> are
> >>> going to be renewed over time to try and avoid a probable memory leak.
> >>> <E0><EF><F0> 05, 2016 4:21:52 PM
> >>> org.apache.catalina.loader.WebappClassLoaderBase
> >>> checkThreadLocalMapForLeaks
> >>> SEVERE: The web application [/app1] created a ThreadLocal with key of
> >>> type
> >>> [org.apache.ignite.internal.binary.BinaryThreadLocalContext$1] (value
> >>> [org.apache.ignite.internal.binary.BinaryThreadLocalContext$1@4e9300e7
> ])
> >>> and a value of type
> >>> [org.apache.ignite.internal.binary.BinaryThreadLocalContext] (value
> >>> [org.apache.ignite.internal.binary.BinaryThreadLocalContext@7934a88])
> >>> but
> >>> failed to remove it when the web application was stopped. Threads are
> >>> going
> >>> to be renewed over time to try and avoid a probable memory leak.
> >>> <E0><EF><F0> 05, 2016 4:21:52 PM
> >>> org.apache.catalina.loader.WebappClassLoaderBase
> >>> checkThreadLocalMapForLeaks
> >>> SEVERE: The web application [/app1] created a ThreadLocal with key of
> >>> type
> >>> [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@56502b72]) and a
> >>> value of type
> >>> [org.apache.ignite.internal.binary.streams.BinaryMemoryAllocatorChunk]
> >>> (value
> >>>
> >>>
> >>>
> [org.apache.ignite.internal.binary.streams.BinaryMemoryAllocatorChunk@4eab9d2f
> >>> ])
> >>> but failed to remove it when the web application was stopped. Threads
> are
> >>> going to be renewed over time to try and avoid a probable memory leak.
> >>> <E0><EF><F0> 05, 2016 4:21:52 PM
> >>> org.apache.catalina.loader.WebappClassLoaderBase
> >>> checkThreadLocalMapForLeaks
> >>> SEVERE: The web application [/app1] created a ThreadLocal with key of
> >>> type
> >>> [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@eb53c9d]) and a
> >>> value
> >>> of type
> >>>
> >>>
> >>>
> [org.apache.ignite.marshaller.optimized.OptimizedObjectStreamRegistry.StreamHolder]
> >>> (value
> >>>
> >>>
> >>>
> [org.apache.ignite.marshaller.optimized.OptimizedObjectStreamRegistry$StreamHolder@5e2c15f7
> >>> ])
> >>> but failed to remove it when the web application was stopped. Threads
> are
> >>> going to be renewed over time to try and avoid a probable memory leak.
> >>> <E0><EF><F0> 05, 2016 4:21:52 PM
> >>> org.apache.catalina.loader.WebappClassLoaderBase
> >>> checkThreadLocalMapForLeaks
> >>> SEVERE: The web application [/app1] created a ThreadLocal with key of
> >>> type
> >>> [org.apache.ignite.internal.util.GridSpinReadWriteLock$1] (value
> >>> [org.apache.ignite.internal.util.GridSpinReadWriteLock$1@55b9da28])
> and
> >>> a
> >>> value of type [java.lang.Integer] (value [0]) but failed to remove it
> >>> when
> >>> the web application was stopped. Threads are going to be renewed over
> >>> time
> >>> to try and avoid a probable memory leak.
> >>> <E0><EF><F0> 05, 2016 4:21:52 PM
> >>> org.apache.catalina.loader.WebappClassLoaderBase
> >>> checkThreadLocalMapForLeaks
> >>> SEVERE: The web application [/app1] created a ThreadLocal with key of
> >>> type
> >>> [org.apache.ignite.internal.util.StripedCompositeReadWriteLock$1]
> (value
> >>>
> [org.apache.ignite.internal.util.StripedCompositeReadWriteLock$1@1ac73431
> >>> ])
> >>> and a value of type [java.lang.Integer] (value [9]) but failed to
> remove
> >>> it
> >>> when the web application was stopped. Threads are going to be renewed
> >>> over
> >>> time to try and avoid a probable memory leak.
> >>> SEVERE: The web application [/app1] created a ThreadLocal with key of
> >>> type
> >>> [org.apache.ignite.internal.processors.cache.GridCacheAdapter$5] (value
> >>>
> [org.apache.ignite.internal.processors.cache.GridCacheAdapter$5@28a1bd99
> >>> ])
> >>> and a value of type
> >>>
> >>>
> [org.apache.ignite.internal.processors.cache.GridCacheAdapter.FutureHolder]
> >>> (value
> >>>
> >>>
> >>>
> [org.apache.ignite.internal.processors.cache.GridCacheAdapter$FutureHolder@6763d70c
> >>> ])
> >>> but failed to remove it when the web application was stopped. Threads
> are
> >>> going to be renewed over time to try and avoid a probable memory leak.
> >>>
> >>> As a solution, I propose to implement servlet filter as a part of
> Ignite
> >>> web integration, which will be responsible for cleaning up all thread
> >>> locals after request processing.
> >>> Filter must be executed last in filter chain to work correctly.
> >>> It is not necessary to use the filter in case of Tomcat, because it
> >>> provides own solution to the problem requiring special configuration
> >>> parameter for context
> >>> User should decide which approach is better to him.
> >>>
> >>>
> >>> 2016-04-06 1:45 GMT+03:00 Valentin Kulichenko <
> >>> valentin.kulichenko@gmail.com
> >>>
> >>>> :
> >>>> Alexei,
> >>>>
> >>>> I would start with defining which thread locals cause this and why
> they
> >>>>
> >>> are
> >>>
> >>>> not properly cleaned on node shutdown. Ideally, this should never
> >>>> happen.
> >>>>
> >>>> -Val
> >>>>
> >>>> On Tue, Apr 5, 2016 at 10:36 AM, Alexei Scherbakov <
> >>>> alexey.scherbakoff@gmail.com> wrote:
> >>>>
> >>>> Hello.
> >>>>>
> >>>>> Currently I'm working on the IGNITE-967
> >>>>> <https://issues.apache.org/jira/browse/IGNITE-967>.
> >>>>>
> >>>>> In fact, it is well known, easily reproduceable, ClassLoader memory
> >>>>>
> >>>> leak
> >>>
> >>>> problem in managed environments, such as servlet containers.
> >>>>> See http://wiki.apache.org/tomcat/MemoryLeakProtection to read more
> on
> >>>>> that.
> >>>>> In short, the code using thread locals is responsible for cleaning
> them
> >>>>> after request processing
> >>>>> to prevent memory leaks related to different lifetime of application
> >>>>>
> >>>> and
> >>>
> >>>> thread in pool.
> >>>>> Also blindly reusing thread from pool can be dangerous, because
> current
> >>>>> ThreadLocal value can be different from expected default value.
> >>>>>
> >>>>> One posiible way of solving the problem is using post-processing
> >>>>>
> >>>> Filter,
> >>>
> >>>> which is responsible for cleaning all thread locals on current thread.
> >>>>>
> >>>>> Another approach would be sticking with container's specific solution
> >>>>>
> >>>> to
> >>>
> >>>> this problem.
> >>>>> On example, tomcat provides the configuration property
> >>>>> renewThreadsWhenStoppingContext,
> >>>>> which forces it to renew threads involved in servicing requests for
> >>>>> destroyed context, thus removing leaks.
> >>>>>
> >>>>> What do you think?
> >>>>>
> >>>>>
> >>>>> --
> >>>>>
> >>>>> Best regards,
> >>>>> Alexei Scherbakov
> >>>>>
> >>>>>
> >>>
> >>> --
> >>>
> >>> Best regards,
> >>> Alexei Scherbakov
> >>>
> >>>
> >
>
>
> --
>
> Best regards,
> Alexei Scherbakov
>

Re: Cleaning internal thread locals

Posted by Alexei Scherbakov <al...@gmail.com>.
It is a very good idea to try to fix internal thread local cleaning in all
places.
On example, it can be easily done in class GridSpinReadWriteLock
Just replace code in line 172 with
readLockEntryCnt.remove();
and add line 388 with:
if (update == 0) readLockEntryCnt.remove();
( needs more testing possibly )

I've provided you with the list of offending places ( which may not be
completed )
But it is definitely a good place to start.
Who will be responsible for fixiing it ?


2016-04-06 11:50 GMT+03:00 Denis Magda <dm...@gridgain.com>:

> As far as I understand Alexei, the issue here is that Ignite kernal
> doesn't have list of all the threads that accessed it during its lifetime.
> So the kernal won't be able to iterate over all the threads (like threads
> from Tomcat thread pool) and perform all the cleaning. This is the issue.
>
> --
> Denis
>
>
> On 4/6/2016 11:47 AM, Artem Shutak wrote:
>
>> Alexei,
>>
>> I would follow the Valentin's suggestion in the issue description and
>> would
>> add to Ignite tests a check on thread locals after stopping all grids. And
>> then we will be able to fix thread local cleaning in all places.
>>
>> Thanks,
>> -- Artem --
>>
>> On Wed, Apr 6, 2016 at 11:18 AM, Alexei Scherbakov <
>> alexey.scherbakoff@gmail.com> wrote:
>>
>> Valentin,
>>>
>>> I did test on demo web app where HttpSession is backed by Ignite cache
>>> and
>>> collect some logs.
>>> There are many places in Ignite where thread locals are used.
>>> Then cache methods are accessed in threads from servlet container pool,
>>> we
>>> get leaks.
>>> I don't think it will be easy to add thread locals cleaning in all these
>>> places without major code rewrite, but, for sure, it would be ideal
>>> solution.
>>>
>>>
>>> <E0><EF><F0> 05, 2016 4:21:52 PM
>>> org.apache.catalina.loader.WebappClassLoaderBase
>>> checkThreadLocalMapForLeaks
>>> SEVERE: The web application [/app1] created a ThreadLocal with key of
>>> type
>>> [org.apache.ignite.internal.util.GridSpinReadWriteLock$1] (value
>>> [org.apache.ignite.internal.util.GridSpinReadWriteLock$1@5fecc35f]) and
>>> a
>>> value of type [java.lang.Integer] (value [0]) but failed to remove it
>>> when
>>> the web application was stopped. Threads are going to be renewed over
>>> time
>>> to try and avoid a probable memory leak.
>>> <E0><EF><F0> 05, 2016 4:21:52 PM
>>> org.apache.catalina.loader.WebappClassLoaderBase
>>> checkThreadLocalMapForLeaks
>>> SEVERE: The web application [/app1] created a ThreadLocal with key of
>>> type
>>> [org.apache.ignite.internal.util.tostring.GridToStringBuilder$1] (value
>>> [org.apache.ignite.internal.util.tostring.GridToStringBuilder$1@4ecdfc04
>>> ])
>>> and a value of type [java.util.LinkedList] (value
>>>
>>> [[org.apache.ignite.internal.util.tostring.GridToStringThreadLocal@684a2791
>>> ]])
>>> but failed to remove it when the web application was stopped. Threads are
>>> going to be renewed over time to try and avoid a probable memory leak.
>>> <E0><EF><F0> 05, 2016 4:21:52 PM
>>> org.apache.catalina.loader.WebappClassLoaderBase
>>> checkThreadLocalMapForLeaks
>>> SEVERE: The web application [/app1] created a ThreadLocal with key of
>>> type
>>> [org.apache.ignite.internal.binary.BinaryThreadLocalContext$1] (value
>>> [org.apache.ignite.internal.binary.BinaryThreadLocalContext$1@4e9300e7])
>>> and a value of type
>>> [org.apache.ignite.internal.binary.BinaryThreadLocalContext] (value
>>> [org.apache.ignite.internal.binary.BinaryThreadLocalContext@7934a88])
>>> but
>>> failed to remove it when the web application was stopped. Threads are
>>> going
>>> to be renewed over time to try and avoid a probable memory leak.
>>> <E0><EF><F0> 05, 2016 4:21:52 PM
>>> org.apache.catalina.loader.WebappClassLoaderBase
>>> checkThreadLocalMapForLeaks
>>> SEVERE: The web application [/app1] created a ThreadLocal with key of
>>> type
>>> [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@56502b72]) and a
>>> value of type
>>> [org.apache.ignite.internal.binary.streams.BinaryMemoryAllocatorChunk]
>>> (value
>>>
>>>
>>> [org.apache.ignite.internal.binary.streams.BinaryMemoryAllocatorChunk@4eab9d2f
>>> ])
>>> but failed to remove it when the web application was stopped. Threads are
>>> going to be renewed over time to try and avoid a probable memory leak.
>>> <E0><EF><F0> 05, 2016 4:21:52 PM
>>> org.apache.catalina.loader.WebappClassLoaderBase
>>> checkThreadLocalMapForLeaks
>>> SEVERE: The web application [/app1] created a ThreadLocal with key of
>>> type
>>> [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@eb53c9d]) and a
>>> value
>>> of type
>>>
>>>
>>> [org.apache.ignite.marshaller.optimized.OptimizedObjectStreamRegistry.StreamHolder]
>>> (value
>>>
>>>
>>> [org.apache.ignite.marshaller.optimized.OptimizedObjectStreamRegistry$StreamHolder@5e2c15f7
>>> ])
>>> but failed to remove it when the web application was stopped. Threads are
>>> going to be renewed over time to try and avoid a probable memory leak.
>>> <E0><EF><F0> 05, 2016 4:21:52 PM
>>> org.apache.catalina.loader.WebappClassLoaderBase
>>> checkThreadLocalMapForLeaks
>>> SEVERE: The web application [/app1] created a ThreadLocal with key of
>>> type
>>> [org.apache.ignite.internal.util.GridSpinReadWriteLock$1] (value
>>> [org.apache.ignite.internal.util.GridSpinReadWriteLock$1@55b9da28]) and
>>> a
>>> value of type [java.lang.Integer] (value [0]) but failed to remove it
>>> when
>>> the web application was stopped. Threads are going to be renewed over
>>> time
>>> to try and avoid a probable memory leak.
>>> <E0><EF><F0> 05, 2016 4:21:52 PM
>>> org.apache.catalina.loader.WebappClassLoaderBase
>>> checkThreadLocalMapForLeaks
>>> SEVERE: The web application [/app1] created a ThreadLocal with key of
>>> type
>>> [org.apache.ignite.internal.util.StripedCompositeReadWriteLock$1] (value
>>> [org.apache.ignite.internal.util.StripedCompositeReadWriteLock$1@1ac73431
>>> ])
>>> and a value of type [java.lang.Integer] (value [9]) but failed to remove
>>> it
>>> when the web application was stopped. Threads are going to be renewed
>>> over
>>> time to try and avoid a probable memory leak.
>>> SEVERE: The web application [/app1] created a ThreadLocal with key of
>>> type
>>> [org.apache.ignite.internal.processors.cache.GridCacheAdapter$5] (value
>>> [org.apache.ignite.internal.processors.cache.GridCacheAdapter$5@28a1bd99
>>> ])
>>> and a value of type
>>>
>>> [org.apache.ignite.internal.processors.cache.GridCacheAdapter.FutureHolder]
>>> (value
>>>
>>>
>>> [org.apache.ignite.internal.processors.cache.GridCacheAdapter$FutureHolder@6763d70c
>>> ])
>>> but failed to remove it when the web application was stopped. Threads are
>>> going to be renewed over time to try and avoid a probable memory leak.
>>>
>>> As a solution, I propose to implement servlet filter as a part of Ignite
>>> web integration, which will be responsible for cleaning up all thread
>>> locals after request processing.
>>> Filter must be executed last in filter chain to work correctly.
>>> It is not necessary to use the filter in case of Tomcat, because it
>>> provides own solution to the problem requiring special configuration
>>> parameter for context
>>> User should decide which approach is better to him.
>>>
>>>
>>> 2016-04-06 1:45 GMT+03:00 Valentin Kulichenko <
>>> valentin.kulichenko@gmail.com
>>>
>>>> :
>>>> Alexei,
>>>>
>>>> I would start with defining which thread locals cause this and why they
>>>>
>>> are
>>>
>>>> not properly cleaned on node shutdown. Ideally, this should never
>>>> happen.
>>>>
>>>> -Val
>>>>
>>>> On Tue, Apr 5, 2016 at 10:36 AM, Alexei Scherbakov <
>>>> alexey.scherbakoff@gmail.com> wrote:
>>>>
>>>> Hello.
>>>>>
>>>>> Currently I'm working on the IGNITE-967
>>>>> <https://issues.apache.org/jira/browse/IGNITE-967>.
>>>>>
>>>>> In fact, it is well known, easily reproduceable, ClassLoader memory
>>>>>
>>>> leak
>>>
>>>> problem in managed environments, such as servlet containers.
>>>>> See http://wiki.apache.org/tomcat/MemoryLeakProtection to read more on
>>>>> that.
>>>>> In short, the code using thread locals is responsible for cleaning them
>>>>> after request processing
>>>>> to prevent memory leaks related to different lifetime of application
>>>>>
>>>> and
>>>
>>>> thread in pool.
>>>>> Also blindly reusing thread from pool can be dangerous, because current
>>>>> ThreadLocal value can be different from expected default value.
>>>>>
>>>>> One posiible way of solving the problem is using post-processing
>>>>>
>>>> Filter,
>>>
>>>> which is responsible for cleaning all thread locals on current thread.
>>>>>
>>>>> Another approach would be sticking with container's specific solution
>>>>>
>>>> to
>>>
>>>> this problem.
>>>>> On example, tomcat provides the configuration property
>>>>> renewThreadsWhenStoppingContext,
>>>>> which forces it to renew threads involved in servicing requests for
>>>>> destroyed context, thus removing leaks.
>>>>>
>>>>> What do you think?
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Best regards,
>>>>> Alexei Scherbakov
>>>>>
>>>>>
>>>
>>> --
>>>
>>> Best regards,
>>> Alexei Scherbakov
>>>
>>>
>


-- 

Best regards,
Alexei Scherbakov

Re: Cleaning internal thread locals

Posted by Denis Magda <dm...@gridgain.com>.
As far as I understand Alexei, the issue here is that Ignite kernal 
doesn't have list of all the threads that accessed it during its lifetime.
So the kernal won't be able to iterate over all the threads (like 
threads from Tomcat thread pool) and perform all the cleaning. This is 
the issue.

--
Denis

On 4/6/2016 11:47 AM, Artem Shutak wrote:
> Alexei,
>
> I would follow the Valentin's suggestion in the issue description and would
> add to Ignite tests a check on thread locals after stopping all grids. And
> then we will be able to fix thread local cleaning in all places.
>
> Thanks,
> -- Artem --
>
> On Wed, Apr 6, 2016 at 11:18 AM, Alexei Scherbakov <
> alexey.scherbakoff@gmail.com> wrote:
>
>> Valentin,
>>
>> I did test on demo web app where HttpSession is backed by Ignite cache and
>> collect some logs.
>> There are many places in Ignite where thread locals are used.
>> Then cache methods are accessed in threads from servlet container pool, we
>> get leaks.
>> I don't think it will be easy to add thread locals cleaning in all these
>> places without major code rewrite, but, for sure, it would be ideal
>> solution.
>>
>>
>> <E0><EF><F0> 05, 2016 4:21:52 PM
>> org.apache.catalina.loader.WebappClassLoaderBase
>> checkThreadLocalMapForLeaks
>> SEVERE: The web application [/app1] created a ThreadLocal with key of type
>> [org.apache.ignite.internal.util.GridSpinReadWriteLock$1] (value
>> [org.apache.ignite.internal.util.GridSpinReadWriteLock$1@5fecc35f]) and a
>> value of type [java.lang.Integer] (value [0]) but failed to remove it when
>> the web application was stopped. Threads are going to be renewed over time
>> to try and avoid a probable memory leak.
>> <E0><EF><F0> 05, 2016 4:21:52 PM
>> org.apache.catalina.loader.WebappClassLoaderBase
>> checkThreadLocalMapForLeaks
>> SEVERE: The web application [/app1] created a ThreadLocal with key of type
>> [org.apache.ignite.internal.util.tostring.GridToStringBuilder$1] (value
>> [org.apache.ignite.internal.util.tostring.GridToStringBuilder$1@4ecdfc04])
>> and a value of type [java.util.LinkedList] (value
>> [[org.apache.ignite.internal.util.tostring.GridToStringThreadLocal@684a2791
>> ]])
>> but failed to remove it when the web application was stopped. Threads are
>> going to be renewed over time to try and avoid a probable memory leak.
>> <E0><EF><F0> 05, 2016 4:21:52 PM
>> org.apache.catalina.loader.WebappClassLoaderBase
>> checkThreadLocalMapForLeaks
>> SEVERE: The web application [/app1] created a ThreadLocal with key of type
>> [org.apache.ignite.internal.binary.BinaryThreadLocalContext$1] (value
>> [org.apache.ignite.internal.binary.BinaryThreadLocalContext$1@4e9300e7])
>> and a value of type
>> [org.apache.ignite.internal.binary.BinaryThreadLocalContext] (value
>> [org.apache.ignite.internal.binary.BinaryThreadLocalContext@7934a88]) but
>> failed to remove it when the web application was stopped. Threads are going
>> to be renewed over time to try and avoid a probable memory leak.
>> <E0><EF><F0> 05, 2016 4:21:52 PM
>> org.apache.catalina.loader.WebappClassLoaderBase
>> checkThreadLocalMapForLeaks
>> SEVERE: The web application [/app1] created a ThreadLocal with key of type
>> [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@56502b72]) and a
>> value of type
>> [org.apache.ignite.internal.binary.streams.BinaryMemoryAllocatorChunk]
>> (value
>>
>> [org.apache.ignite.internal.binary.streams.BinaryMemoryAllocatorChunk@4eab9d2f
>> ])
>> but failed to remove it when the web application was stopped. Threads are
>> going to be renewed over time to try and avoid a probable memory leak.
>> <E0><EF><F0> 05, 2016 4:21:52 PM
>> org.apache.catalina.loader.WebappClassLoaderBase
>> checkThreadLocalMapForLeaks
>> SEVERE: The web application [/app1] created a ThreadLocal with key of type
>> [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@eb53c9d]) and a
>> value
>> of type
>>
>> [org.apache.ignite.marshaller.optimized.OptimizedObjectStreamRegistry.StreamHolder]
>> (value
>>
>> [org.apache.ignite.marshaller.optimized.OptimizedObjectStreamRegistry$StreamHolder@5e2c15f7
>> ])
>> but failed to remove it when the web application was stopped. Threads are
>> going to be renewed over time to try and avoid a probable memory leak.
>> <E0><EF><F0> 05, 2016 4:21:52 PM
>> org.apache.catalina.loader.WebappClassLoaderBase
>> checkThreadLocalMapForLeaks
>> SEVERE: The web application [/app1] created a ThreadLocal with key of type
>> [org.apache.ignite.internal.util.GridSpinReadWriteLock$1] (value
>> [org.apache.ignite.internal.util.GridSpinReadWriteLock$1@55b9da28]) and a
>> value of type [java.lang.Integer] (value [0]) but failed to remove it when
>> the web application was stopped. Threads are going to be renewed over time
>> to try and avoid a probable memory leak.
>> <E0><EF><F0> 05, 2016 4:21:52 PM
>> org.apache.catalina.loader.WebappClassLoaderBase
>> checkThreadLocalMapForLeaks
>> SEVERE: The web application [/app1] created a ThreadLocal with key of type
>> [org.apache.ignite.internal.util.StripedCompositeReadWriteLock$1] (value
>> [org.apache.ignite.internal.util.StripedCompositeReadWriteLock$1@1ac73431
>> ])
>> and a value of type [java.lang.Integer] (value [9]) but failed to remove it
>> when the web application was stopped. Threads are going to be renewed over
>> time to try and avoid a probable memory leak.
>> SEVERE: The web application [/app1] created a ThreadLocal with key of type
>> [org.apache.ignite.internal.processors.cache.GridCacheAdapter$5] (value
>> [org.apache.ignite.internal.processors.cache.GridCacheAdapter$5@28a1bd99])
>> and a value of type
>> [org.apache.ignite.internal.processors.cache.GridCacheAdapter.FutureHolder]
>> (value
>>
>> [org.apache.ignite.internal.processors.cache.GridCacheAdapter$FutureHolder@6763d70c
>> ])
>> but failed to remove it when the web application was stopped. Threads are
>> going to be renewed over time to try and avoid a probable memory leak.
>>
>> As a solution, I propose to implement servlet filter as a part of Ignite
>> web integration, which will be responsible for cleaning up all thread
>> locals after request processing.
>> Filter must be executed last in filter chain to work correctly.
>> It is not necessary to use the filter in case of Tomcat, because it
>> provides own solution to the problem requiring special configuration
>> parameter for context
>> User should decide which approach is better to him.
>>
>>
>> 2016-04-06 1:45 GMT+03:00 Valentin Kulichenko <
>> valentin.kulichenko@gmail.com
>>> :
>>> Alexei,
>>>
>>> I would start with defining which thread locals cause this and why they
>> are
>>> not properly cleaned on node shutdown. Ideally, this should never happen.
>>>
>>> -Val
>>>
>>> On Tue, Apr 5, 2016 at 10:36 AM, Alexei Scherbakov <
>>> alexey.scherbakoff@gmail.com> wrote:
>>>
>>>> Hello.
>>>>
>>>> Currently I'm working on the IGNITE-967
>>>> <https://issues.apache.org/jira/browse/IGNITE-967>.
>>>>
>>>> In fact, it is well known, easily reproduceable, ClassLoader memory
>> leak
>>>> problem in managed environments, such as servlet containers.
>>>> See http://wiki.apache.org/tomcat/MemoryLeakProtection to read more on
>>>> that.
>>>> In short, the code using thread locals is responsible for cleaning them
>>>> after request processing
>>>> to prevent memory leaks related to different lifetime of application
>> and
>>>> thread in pool.
>>>> Also blindly reusing thread from pool can be dangerous, because current
>>>> ThreadLocal value can be different from expected default value.
>>>>
>>>> One posiible way of solving the problem is using post-processing
>> Filter,
>>>> which is responsible for cleaning all thread locals on current thread.
>>>>
>>>> Another approach would be sticking with container's specific solution
>> to
>>>> this problem.
>>>> On example, tomcat provides the configuration property
>>>> renewThreadsWhenStoppingContext,
>>>> which forces it to renew threads involved in servicing requests for
>>>> destroyed context, thus removing leaks.
>>>>
>>>> What do you think?
>>>>
>>>>
>>>> --
>>>>
>>>> Best regards,
>>>> Alexei Scherbakov
>>>>
>>
>>
>> --
>>
>> Best regards,
>> Alexei Scherbakov
>>

Re: Cleaning internal thread locals

Posted by Artem Shutak <as...@gridgain.com>.
Alexei,

I would follow the Valentin's suggestion in the issue description and would
add to Ignite tests a check on thread locals after stopping all grids. And
then we will be able to fix thread local cleaning in all places.

Thanks,
-- Artem --

On Wed, Apr 6, 2016 at 11:18 AM, Alexei Scherbakov <
alexey.scherbakoff@gmail.com> wrote:

> Valentin,
>
> I did test on demo web app where HttpSession is backed by Ignite cache and
> collect some logs.
> There are many places in Ignite where thread locals are used.
> Then cache methods are accessed in threads from servlet container pool, we
> get leaks.
> I don't think it will be easy to add thread locals cleaning in all these
> places without major code rewrite, but, for sure, it would be ideal
> solution.
>
>
> <E0><EF><F0> 05, 2016 4:21:52 PM
> org.apache.catalina.loader.WebappClassLoaderBase
> checkThreadLocalMapForLeaks
> SEVERE: The web application [/app1] created a ThreadLocal with key of type
> [org.apache.ignite.internal.util.GridSpinReadWriteLock$1] (value
> [org.apache.ignite.internal.util.GridSpinReadWriteLock$1@5fecc35f]) and a
> value of type [java.lang.Integer] (value [0]) but failed to remove it when
> the web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> <E0><EF><F0> 05, 2016 4:21:52 PM
> org.apache.catalina.loader.WebappClassLoaderBase
> checkThreadLocalMapForLeaks
> SEVERE: The web application [/app1] created a ThreadLocal with key of type
> [org.apache.ignite.internal.util.tostring.GridToStringBuilder$1] (value
> [org.apache.ignite.internal.util.tostring.GridToStringBuilder$1@4ecdfc04])
> and a value of type [java.util.LinkedList] (value
> [[org.apache.ignite.internal.util.tostring.GridToStringThreadLocal@684a2791
> ]])
> but failed to remove it when the web application was stopped. Threads are
> going to be renewed over time to try and avoid a probable memory leak.
> <E0><EF><F0> 05, 2016 4:21:52 PM
> org.apache.catalina.loader.WebappClassLoaderBase
> checkThreadLocalMapForLeaks
> SEVERE: The web application [/app1] created a ThreadLocal with key of type
> [org.apache.ignite.internal.binary.BinaryThreadLocalContext$1] (value
> [org.apache.ignite.internal.binary.BinaryThreadLocalContext$1@4e9300e7])
> and a value of type
> [org.apache.ignite.internal.binary.BinaryThreadLocalContext] (value
> [org.apache.ignite.internal.binary.BinaryThreadLocalContext@7934a88]) but
> failed to remove it when the web application was stopped. Threads are going
> to be renewed over time to try and avoid a probable memory leak.
> <E0><EF><F0> 05, 2016 4:21:52 PM
> org.apache.catalina.loader.WebappClassLoaderBase
> checkThreadLocalMapForLeaks
> SEVERE: The web application [/app1] created a ThreadLocal with key of type
> [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@56502b72]) and a
> value of type
> [org.apache.ignite.internal.binary.streams.BinaryMemoryAllocatorChunk]
> (value
>
> [org.apache.ignite.internal.binary.streams.BinaryMemoryAllocatorChunk@4eab9d2f
> ])
> but failed to remove it when the web application was stopped. Threads are
> going to be renewed over time to try and avoid a probable memory leak.
> <E0><EF><F0> 05, 2016 4:21:52 PM
> org.apache.catalina.loader.WebappClassLoaderBase
> checkThreadLocalMapForLeaks
> SEVERE: The web application [/app1] created a ThreadLocal with key of type
> [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@eb53c9d]) and a
> value
> of type
>
> [org.apache.ignite.marshaller.optimized.OptimizedObjectStreamRegistry.StreamHolder]
> (value
>
> [org.apache.ignite.marshaller.optimized.OptimizedObjectStreamRegistry$StreamHolder@5e2c15f7
> ])
> but failed to remove it when the web application was stopped. Threads are
> going to be renewed over time to try and avoid a probable memory leak.
> <E0><EF><F0> 05, 2016 4:21:52 PM
> org.apache.catalina.loader.WebappClassLoaderBase
> checkThreadLocalMapForLeaks
> SEVERE: The web application [/app1] created a ThreadLocal with key of type
> [org.apache.ignite.internal.util.GridSpinReadWriteLock$1] (value
> [org.apache.ignite.internal.util.GridSpinReadWriteLock$1@55b9da28]) and a
> value of type [java.lang.Integer] (value [0]) but failed to remove it when
> the web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> <E0><EF><F0> 05, 2016 4:21:52 PM
> org.apache.catalina.loader.WebappClassLoaderBase
> checkThreadLocalMapForLeaks
> SEVERE: The web application [/app1] created a ThreadLocal with key of type
> [org.apache.ignite.internal.util.StripedCompositeReadWriteLock$1] (value
> [org.apache.ignite.internal.util.StripedCompositeReadWriteLock$1@1ac73431
> ])
> and a value of type [java.lang.Integer] (value [9]) but failed to remove it
> when the web application was stopped. Threads are going to be renewed over
> time to try and avoid a probable memory leak.
> SEVERE: The web application [/app1] created a ThreadLocal with key of type
> [org.apache.ignite.internal.processors.cache.GridCacheAdapter$5] (value
> [org.apache.ignite.internal.processors.cache.GridCacheAdapter$5@28a1bd99])
> and a value of type
> [org.apache.ignite.internal.processors.cache.GridCacheAdapter.FutureHolder]
> (value
>
> [org.apache.ignite.internal.processors.cache.GridCacheAdapter$FutureHolder@6763d70c
> ])
> but failed to remove it when the web application was stopped. Threads are
> going to be renewed over time to try and avoid a probable memory leak.
>
> As a solution, I propose to implement servlet filter as a part of Ignite
> web integration, which will be responsible for cleaning up all thread
> locals after request processing.
> Filter must be executed last in filter chain to work correctly.
> It is not necessary to use the filter in case of Tomcat, because it
> provides own solution to the problem requiring special configuration
> parameter for context
> User should decide which approach is better to him.
>
>
> 2016-04-06 1:45 GMT+03:00 Valentin Kulichenko <
> valentin.kulichenko@gmail.com
> >:
>
> > Alexei,
> >
> > I would start with defining which thread locals cause this and why they
> are
> > not properly cleaned on node shutdown. Ideally, this should never happen.
> >
> > -Val
> >
> > On Tue, Apr 5, 2016 at 10:36 AM, Alexei Scherbakov <
> > alexey.scherbakoff@gmail.com> wrote:
> >
> > > Hello.
> > >
> > > Currently I'm working on the IGNITE-967
> > > <https://issues.apache.org/jira/browse/IGNITE-967>.
> > >
> > > In fact, it is well known, easily reproduceable, ClassLoader memory
> leak
> > > problem in managed environments, such as servlet containers.
> > > See http://wiki.apache.org/tomcat/MemoryLeakProtection to read more on
> > > that.
> > > In short, the code using thread locals is responsible for cleaning them
> > > after request processing
> > > to prevent memory leaks related to different lifetime of application
> and
> > > thread in pool.
> > > Also blindly reusing thread from pool can be dangerous, because current
> > > ThreadLocal value can be different from expected default value.
> > >
> > > One posiible way of solving the problem is using post-processing
> Filter,
> > > which is responsible for cleaning all thread locals on current thread.
> > >
> > > Another approach would be sticking with container's specific solution
> to
> > > this problem.
> > > On example, tomcat provides the configuration property
> > > renewThreadsWhenStoppingContext,
> > > which forces it to renew threads involved in servicing requests for
> > > destroyed context, thus removing leaks.
> > >
> > > What do you think?
> > >
> > >
> > > --
> > >
> > > Best regards,
> > > Alexei Scherbakov
> > >
> >
>
>
>
> --
>
> Best regards,
> Alexei Scherbakov
>

Re: Cleaning internal thread locals

Posted by Alexei Scherbakov <al...@gmail.com>.
Valentin,

I did test on demo web app where HttpSession is backed by Ignite cache and
collect some logs.
There are many places in Ignite where thread locals are used.
Then cache methods are accessed in threads from servlet container pool, we
get leaks.
I don't think it will be easy to add thread locals cleaning in all these
places without major code rewrite, but, for sure, it would be ideal
solution.


<E0><EF><F0> 05, 2016 4:21:52 PM
org.apache.catalina.loader.WebappClassLoaderBase checkThreadLocalMapForLeaks
SEVERE: The web application [/app1] created a ThreadLocal with key of type
[org.apache.ignite.internal.util.GridSpinReadWriteLock$1] (value
[org.apache.ignite.internal.util.GridSpinReadWriteLock$1@5fecc35f]) and a
value of type [java.lang.Integer] (value [0]) but failed to remove it when
the web application was stopped. Threads are going to be renewed over time
to try and avoid a probable memory leak.
<E0><EF><F0> 05, 2016 4:21:52 PM
org.apache.catalina.loader.WebappClassLoaderBase checkThreadLocalMapForLeaks
SEVERE: The web application [/app1] created a ThreadLocal with key of type
[org.apache.ignite.internal.util.tostring.GridToStringBuilder$1] (value
[org.apache.ignite.internal.util.tostring.GridToStringBuilder$1@4ecdfc04])
and a value of type [java.util.LinkedList] (value
[[org.apache.ignite.internal.util.tostring.GridToStringThreadLocal@684a2791]])
but failed to remove it when the web application was stopped. Threads are
going to be renewed over time to try and avoid a probable memory leak.
<E0><EF><F0> 05, 2016 4:21:52 PM
org.apache.catalina.loader.WebappClassLoaderBase checkThreadLocalMapForLeaks
SEVERE: The web application [/app1] created a ThreadLocal with key of type
[org.apache.ignite.internal.binary.BinaryThreadLocalContext$1] (value
[org.apache.ignite.internal.binary.BinaryThreadLocalContext$1@4e9300e7])
and a value of type
[org.apache.ignite.internal.binary.BinaryThreadLocalContext] (value
[org.apache.ignite.internal.binary.BinaryThreadLocalContext@7934a88]) but
failed to remove it when the web application was stopped. Threads are going
to be renewed over time to try and avoid a probable memory leak.
<E0><EF><F0> 05, 2016 4:21:52 PM
org.apache.catalina.loader.WebappClassLoaderBase checkThreadLocalMapForLeaks
SEVERE: The web application [/app1] created a ThreadLocal with key of type
[java.lang.ThreadLocal] (value [java.lang.ThreadLocal@56502b72]) and a
value of type
[org.apache.ignite.internal.binary.streams.BinaryMemoryAllocatorChunk]
(value
[org.apache.ignite.internal.binary.streams.BinaryMemoryAllocatorChunk@4eab9d2f])
but failed to remove it when the web application was stopped. Threads are
going to be renewed over time to try and avoid a probable memory leak.
<E0><EF><F0> 05, 2016 4:21:52 PM
org.apache.catalina.loader.WebappClassLoaderBase checkThreadLocalMapForLeaks
SEVERE: The web application [/app1] created a ThreadLocal with key of type
[java.lang.ThreadLocal] (value [java.lang.ThreadLocal@eb53c9d]) and a value
of type
[org.apache.ignite.marshaller.optimized.OptimizedObjectStreamRegistry.StreamHolder]
(value
[org.apache.ignite.marshaller.optimized.OptimizedObjectStreamRegistry$StreamHolder@5e2c15f7])
but failed to remove it when the web application was stopped. Threads are
going to be renewed over time to try and avoid a probable memory leak.
<E0><EF><F0> 05, 2016 4:21:52 PM
org.apache.catalina.loader.WebappClassLoaderBase checkThreadLocalMapForLeaks
SEVERE: The web application [/app1] created a ThreadLocal with key of type
[org.apache.ignite.internal.util.GridSpinReadWriteLock$1] (value
[org.apache.ignite.internal.util.GridSpinReadWriteLock$1@55b9da28]) and a
value of type [java.lang.Integer] (value [0]) but failed to remove it when
the web application was stopped. Threads are going to be renewed over time
to try and avoid a probable memory leak.
<E0><EF><F0> 05, 2016 4:21:52 PM
org.apache.catalina.loader.WebappClassLoaderBase checkThreadLocalMapForLeaks
SEVERE: The web application [/app1] created a ThreadLocal with key of type
[org.apache.ignite.internal.util.StripedCompositeReadWriteLock$1] (value
[org.apache.ignite.internal.util.StripedCompositeReadWriteLock$1@1ac73431])
and a value of type [java.lang.Integer] (value [9]) but failed to remove it
when the web application was stopped. Threads are going to be renewed over
time to try and avoid a probable memory leak.
SEVERE: The web application [/app1] created a ThreadLocal with key of type
[org.apache.ignite.internal.processors.cache.GridCacheAdapter$5] (value
[org.apache.ignite.internal.processors.cache.GridCacheAdapter$5@28a1bd99])
and a value of type
[org.apache.ignite.internal.processors.cache.GridCacheAdapter.FutureHolder]
(value
[org.apache.ignite.internal.processors.cache.GridCacheAdapter$FutureHolder@6763d70c])
but failed to remove it when the web application was stopped. Threads are
going to be renewed over time to try and avoid a probable memory leak.

As a solution, I propose to implement servlet filter as a part of Ignite
web integration, which will be responsible for cleaning up all thread
locals after request processing.
Filter must be executed last in filter chain to work correctly.
It is not necessary to use the filter in case of Tomcat, because it
provides own solution to the problem requiring special configuration
parameter for context
User should decide which approach is better to him.


2016-04-06 1:45 GMT+03:00 Valentin Kulichenko <valentin.kulichenko@gmail.com
>:

> Alexei,
>
> I would start with defining which thread locals cause this and why they are
> not properly cleaned on node shutdown. Ideally, this should never happen.
>
> -Val
>
> On Tue, Apr 5, 2016 at 10:36 AM, Alexei Scherbakov <
> alexey.scherbakoff@gmail.com> wrote:
>
> > Hello.
> >
> > Currently I'm working on the IGNITE-967
> > <https://issues.apache.org/jira/browse/IGNITE-967>.
> >
> > In fact, it is well known, easily reproduceable, ClassLoader memory leak
> > problem in managed environments, such as servlet containers.
> > See http://wiki.apache.org/tomcat/MemoryLeakProtection to read more on
> > that.
> > In short, the code using thread locals is responsible for cleaning them
> > after request processing
> > to prevent memory leaks related to different lifetime of application and
> > thread in pool.
> > Also blindly reusing thread from pool can be dangerous, because current
> > ThreadLocal value can be different from expected default value.
> >
> > One posiible way of solving the problem is using post-processing Filter,
> > which is responsible for cleaning all thread locals on current thread.
> >
> > Another approach would be sticking with container's specific solution to
> > this problem.
> > On example, tomcat provides the configuration property
> > renewThreadsWhenStoppingContext,
> > which forces it to renew threads involved in servicing requests for
> > destroyed context, thus removing leaks.
> >
> > What do you think?
> >
> >
> > --
> >
> > Best regards,
> > Alexei Scherbakov
> >
>



-- 

Best regards,
Alexei Scherbakov

Re: Cleaning internal thread locals

Posted by Valentin Kulichenko <va...@gmail.com>.
Alexei,

I would start with defining which thread locals cause this and why they are
not properly cleaned on node shutdown. Ideally, this should never happen.

-Val

On Tue, Apr 5, 2016 at 10:36 AM, Alexei Scherbakov <
alexey.scherbakoff@gmail.com> wrote:

> Hello.
>
> Currently I'm working on the IGNITE-967
> <https://issues.apache.org/jira/browse/IGNITE-967>.
>
> In fact, it is well known, easily reproduceable, ClassLoader memory leak
> problem in managed environments, such as servlet containers.
> See http://wiki.apache.org/tomcat/MemoryLeakProtection to read more on
> that.
> In short, the code using thread locals is responsible for cleaning them
> after request processing
> to prevent memory leaks related to different lifetime of application and
> thread in pool.
> Also blindly reusing thread from pool can be dangerous, because current
> ThreadLocal value can be different from expected default value.
>
> One posiible way of solving the problem is using post-processing Filter,
> which is responsible for cleaning all thread locals on current thread.
>
> Another approach would be sticking with container's specific solution to
> this problem.
> On example, tomcat provides the configuration property
> renewThreadsWhenStoppingContext,
> which forces it to renew threads involved in servicing requests for
> destroyed context, thus removing leaks.
>
> What do you think?
>
>
> --
>
> Best regards,
> Alexei Scherbakov
>