[Resolved] clearReferencesThreads, Poller SunPKCS11-Solaris and strange context class loader

[Rainer Jung <ra...@kippdata.de>]

On 06.05.2010 20:51, Rainer Jung wrote:
> While doing some testing with 6.0.26 I noticed, that when shutting it
> down it logs an error about thread "Poller SunPKCS11-Solaris" not being
> stopped. When I add the more explicit logging from tc6 trunk, it says
> the thread was started by /manager.
>
> The thread sits in
>
> java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at sun.security.pkcs11.SunPKCS11$TokenPoller.run(SunPKCS11.java:681)
> at java.lang.Thread.run(Thread.java:619)
>
> and is started for some version of the JDK (noted on Solaris for e.g.
> 1.5.0_22 and 1.6.0_3, not for 1.6.0_17) even without having anything
> related to keystores, https or similar configured in Tomcat (default
> config). The only non-default is using Log4J instead of Juli.
>
> What is strange, is that the check whether the context class loader of
> the thread is equal to the WebappClassLoader of the context to unload
> passes. I added an additional output by explicitely printing the
> contextName of the two loaders and in fact both print "/manager".
>
> When deploying two instances of the manager and reloading the original
> instance with the additional one, I get the same warning.
>
> In a heap dump, it seems the context class loader of the thread is the
> system class loader and not of type WebappClassLoader. Any idea, why the
> context class loader test fails? Is there any reason why the context
> class loader of the thread might change when doing the manager reload or
> shutdown?

1) Why is the class loader in the heap dump not the WebappClassLoader?

For producing a heap dump, I added 
"-Dcom.sun.management.jmxremote=true". As a side effect java.security 
got initialized much earlier and before Tomcat startup happened. So the 
poller thread was started earlier and had the system class loader as 
TCCL. I noticed that by adding "-verbose:class" to track class loading.

2) What triggers startup of that thread in normal Tomcat?

When *not* using the jmx system properties, the thread starts during 
webapp deployment. StandardManager inside start() initializes random 
numbers by calling generateSessionId(), and that method in ManagerBase 
calls getDigest(), which calls java.security.MessageDigest.getInstance() 
which finally initializes the Java Security Providers and starts the 
poller thread.

I didn't actually check it, but I expect the TCCL of the main thread 
during that stage of webapp deployment is the webapp class loader.

If I don't deploy the manager webapp, then the poller is simply started 
during the ROOT webapp deployment. Whatever context is deployed first 
starts the thread.

3) Why isn't it there for all Java versions?

Sun applied a fix to the SunPKCS11 class for the so-called soft token. 
The thread is now started only, if the token is "removable" and the soft 
token is not. The patch is not part e.g. of old Java 6 versions.

So for older JDK on Solaris 10, there is always a so-called soft token, 
and initializing the Java security providers will launch the thread. For 
older Solaris (e.g. 8) there is no soft token and thus no thread. For 
newer JDK patch versions, the soft token is identified to not be 
removable and again no poller thread is started.

4) Do we have to fix anything?

If users do not actually use a removable token, upgrading the JDK is 
enough. Once users start to complain about the leak warning when using a 
removable token, it would suffice to initialize the Java security 
providers before deploying the webapps, so that the thread has the 
system or catalina loader as TCCL. This can be done e.g. by calling 
java.security.Security.getProviders(). But actually I'm not sure, 
whether the warning is a false positive. It's quite possible, that the 
poller thread actually does imply a perm gen leak.

Thanks to all who provided valuable pointers.

Regards,

Rainer

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org