[jira] [Commented] (IGNITE-967) Internal thread locals are not always cleaned

["Alexei Scherbakov (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IGNITE-967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15230140#comment-15230140 ] 

Alexei Scherbakov commented on IGNITE-967:
------------------------------------------

Where are two probable types of leaks induced by Ignite's heavy thread locals usage (required to achive high performance) on webapp context undeployment: 
1) Static thread locals referencing webapp class instances. If such entries are not removed from thread local map, they prevent class instances of undeployed app being GC'ed.
2) Deferred removal of GC'ed entries in thread locals map. Stale entries are removed only when thread locals are accessed, and if the thread remains dormant in the pool, the reference to undeployed webapp remains active. See Oracle JDK implementation of thread locals for details.

It turns out what major AS vendors, such as Tomcat, IBM(WebSphere), JBoss(Wildfly), have build in tools for dealing with the problem by renewing threads in the pool and automatically clearing
static variables on undeploy.
See 
http://wiki.apache.org/tomcat/MemoryLeakProtection 
https://www.ibm.com/support/knowledgecenter/was_beta/com.ibm.websphere.base.doc/ae/ctrb_memleakdetection.html
for more information.
So, I suppose, it would be safe for now to treat these messages as harmless.
There is also the discussion on the topic in Ignite dev list: Cleaning internal thread locals.


> Internal thread locals are not always cleaned
> ---------------------------------------------
>
>                 Key: IGNITE-967
>                 URL: https://issues.apache.org/jira/browse/IGNITE-967
>             Project: Ignite
>          Issue Type: Bug
>          Components: general
>    Affects Versions: sprint-4
>            Reporter: Valentin Kulichenko
>            Assignee: Alexei Scherbakov
>            Priority: Critical
>              Labels: important
>             Fix For: 1.6
>
>
> One of our users reported that he sees warnings in Tomcat's log when the application that's running Ignite in embedded mode is undeployed:
> {code}
> SEVERE: The web application [/XXX] created a ThreadLocal with key of type [org.apache.ignite.internal.util.GridSpinReadWriteLock$1] (value [org.apache.ignite.internal.util.GridSpinReadWriteLock$1@2c2858af]) and a value of type [java.lang.Integer] (value [0]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
> {code}
> There is also the similar warning for {{GridToStringBuilder.threadCache}}. 
> While it's usually OK not to clean thread locals on standalone node, in app server it can cause a memory leak.
> To avoid such issues I suggest to add a special step after all test suites that will check thread locals in test runner thread. If we have this check in CI, we will fix it once and for always.
> Thread local values can be introspected through {{Thread.threadLocals}} variable. It would also be a good idea to check Tomcat's sources on how it's done there.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)