You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tomcat.apache.org by Rainer Jung <ra...@kippdata.de> on 2010/05/06 20:51:22 UTC
clearReferencesThreads, Poller SunPKCS11-Solaris and strange context
class loader
While doing some testing with 6.0.26 I noticed, that when shutting it
down it logs an error about thread "Poller SunPKCS11-Solaris" not being
stopped. When I add the more explicit logging from tc6 trunk, it says
the thread was started by /manager.
The thread sits in
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at sun.security.pkcs11.SunPKCS11$TokenPoller.run(SunPKCS11.java:681)
at java.lang.Thread.run(Thread.java:619)
and is started for some version of the JDK (noted on Solaris for e.g.
1.5.0_22 and 1.6.0_3, not for 1.6.0_17) even without having anything
related to keystores, https or similar configured in Tomcat (default
config). The only non-default is using Log4J instead of Juli.
What is strange, is that the check whether the context class loader of
the thread is equal to the WebappClassLoader of the context to unload
passes. I added an additional output by explicitely printing the
contextName of the two loaders and in fact both print "/manager".
When deploying two instances of the manager and reloading the original
instance with the additional one, I get the same warning.
In a heap dump, it seems the context class loader of the thread is the
system class loader and not of type WebappClassLoader. Any idea, why the
context class loader test fails? Is there any reason why the context
class loader of the thread might change when doing the manager reload or
shutdown?
Regards,
Rainer
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: clearReferencesThreads, Poller SunPKCS11-Solaris and strange
context class loader
Posted by Rainer Jung <ra...@kippdata.de>.
Hi Konstantin,
On 09.05.2010 11:38, Konstantin Kolinko wrote:
> 2010/5/9 Rainer Jung<ra...@kippdata.de>:
>> On 07.05.2010 20:08, Filip Hanik - Dev Lists wrote:
>>>
>>> On 05/07/2010 08:03 AM, Rainer Jung wrote:
>>>>
>>>> It's there after JDK start with some version of the JDK. It's not
>>>> there if I only start a plain Java test doing only a sleep using the
>>>> same JDK.
>>>
>>> well, if the app uses java.net.URL to a https, then this would make
>>> sense that the thread got started with the webapp's class loader
>>
>> Yes, but there is no app. The thread is there even when I remove all webapps
>> (except for the ROOT context I kept). I guess I need to do more systematic
>> testing.
>>
>
> Some more research,
> looking at the source code that I mentioned:
>
> 1. "SunPKCS11-Solaris" is not the default name. The default name of
> the class instance would be "SunPKCS11-Dummy".
>
> You may want to look at the "/lib/security/java.security" file in your
> JRE. It should be configured there.
Yes, I knew. PKCS is in there, but it is by default. The file hasn't
been touched.
> Here is some, though failing. example of trying to use this class:
> http://forums.sun.com/thread.jspa?threadID=5431010
>
> 2. sun.security.pkcs11.SunPKCS11 extends java.security.AuthProvider
>
> 3. SunPKCS11() constructor calls initToken(slotInfo) and that method
> will call createPoller() if the token is not present in the slot.
> (Thus apparently it starts waiting for the token).
>
> 4. It is said that PKCS11 "defines a platform-independent API to
> cryptographic tokens, such as Hardware Security Modules (HSM) and
> smart cards."
The point is:
- there is no cryptographic hardware involved in my case
- Tomcat default configuration used, HTTPS connector disabled
- no tc native
- I don't want to use PKCS or any other crypto
Nevertheless the threads show up and is spotted by the memory leak
detection during manager shutdown/restart as belonging to the manager
webapp (by means of the TCCL), although in the dump before the restart
its TCCL was the system class loader and the thread also gets started
when no manager gets deployed.
My first priority is to remove the false positive for the leak
detection, only the second prio is getting rid of the thread.
Regards,
Rainer
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: clearReferencesThreads, Poller SunPKCS11-Solaris and strange
context class loader
Posted by Konstantin Kolinko <kn...@gmail.com>.
2010/5/9 Rainer Jung <ra...@kippdata.de>:
> On 07.05.2010 20:08, Filip Hanik - Dev Lists wrote:
>>
>> On 05/07/2010 08:03 AM, Rainer Jung wrote:
>>>
>>> It's there after JDK start with some version of the JDK. It's not
>>> there if I only start a plain Java test doing only a sleep using the
>>> same JDK.
>>
>> well, if the app uses java.net.URL to a https, then this would make
>> sense that the thread got started with the webapp's class loader
>
> Yes, but there is no app. The thread is there even when I remove all webapps
> (except for the ROOT context I kept). I guess I need to do more systematic
> testing.
>
Some more research,
looking at the source code that I mentioned:
1. "SunPKCS11-Solaris" is not the default name. The default name of
the class instance would be "SunPKCS11-Dummy".
You may want to look at the "/lib/security/java.security" file in your
JRE. It should be configured there.
Here is some, though failing. example of trying to use this class:
http://forums.sun.com/thread.jspa?threadID=5431010
2. sun.security.pkcs11.SunPKCS11 extends java.security.AuthProvider
3. SunPKCS11() constructor calls initToken(slotInfo) and that method
will call createPoller() if the token is not present in the slot.
(Thus apparently it starts waiting for the token).
4. It is said that PKCS11 "defines a platform-independent API to
cryptographic tokens, such as Hardware Security Modules (HSM) and
smart cards."
Best regards,
Konstantin Kolinko
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: clearReferencesThreads, Poller SunPKCS11-Solaris and strange
context class loader
Posted by Rainer Jung <ra...@kippdata.de>.
On 07.05.2010 20:08, Filip Hanik - Dev Lists wrote:
> On 05/07/2010 08:03 AM, Rainer Jung wrote:
>> It's there after JDK start with some version of the JDK. It's not
>> there if I only start a plain Java test doing only a sleep using the
>> same JDK.
>
> well, if the app uses java.net.URL to a https, then this would make
> sense that the thread got started with the webapp's class loader
Yes, but there is no app. The thread is there even when I remove all
webapps (except for the ROOT context I kept). I guess I need to do more
systematic testing.
Regards,
Rainer
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: clearReferencesThreads, Poller SunPKCS11-Solaris and strange
context class loader
Posted by Filip Hanik - Dev Lists <de...@hanik.com>.
On 05/07/2010 08:03 AM, Rainer Jung wrote:
> On 07.05.2010 14:23, Konstantin Kolinko wrote:
>> 2010/5/7 Rainer Jung<ra...@kippdata.de>:
>>> On 07.05.2010 11:00, Konstantin Kolinko wrote:
>>>>
>>>> 2010/5/7 Rainer Jung<ra...@kippdata.de>:
>>>>>
>>>>> I'm wondering why the PCKS Token
>>>>> Poller thread was captured by the leak prevention. Since we know the
>>>>> code,
>>>>> it was because its context class loader was equal to the
>>>>> WebappClassLoader
>>>>> of /manager. That's what I don't understand. See my original post.
>>>>>
>>>>
>>>> TCCL is inherited from the parent thread. It means that that thread
>>>> was started either during request processing, or during webapplication
>>>> startup.
>>>
>>> But unfortunately that doesn't provide consistent results.
>>>
>>> If I sart Tomcat without any webapp (but with ROOT context) the
>>> thread is
>>> started too. The heap dump shows the system cl as the tccl, but during
>>> restart or shutdown the leak prevention retrieves the tccl and finds
>>> it s
>>> equal to the WebappClassLoader of the manager.
>>>
>>> I'm confused.
>>>
>>>> BTW, 4th result in
>>>> http://www.google.com/search?q=SunPKCS11+TokenPoller
>>>> gives source code for that class.
>>>
>>> I'll see, but I guess the point is more about who starts it when and
>>> why,
>>> less about what it does.
>>>
>>
>> What is this all about, I means this PKCS11 tokens?
>> Is it something that is used in your configuration?
>
> No, not at all. There's not HTTPS connector, no keystore no nothing.
> The PKCS11 provider comes with the JDK but I'm not aware of anything
> that uses it. Default Tomcat configuration plus log4j.
>
> It's there after JDK start with some version of the JDK. It's not
> there if I only start a plain Java test doing only a sleep using the
> same JDK.
well, if the app uses java.net.URL to a https, then this would make
sense that the thread got started with the webapp's class loader
>
>> I mean, what are those "tokens" that it works with? (for a newbie)
>> Do you use JAAS?
>
> Not that I'm aware of.
>
>> From the code, e.g.
>> SunPKCS11#uninitToken() calls
>> 1) destroyPoller()
>> 2) createPoller()
>>
>> which restarts the thread. So the one at shutdown time might be
>> different from the one that you see when Tomcat starts.
>
> Hmmm. Will dig deeper over the weekend.
>
> Regards,
>
> Rainer
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: dev-help@tomcat.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: clearReferencesThreads, Poller SunPKCS11-Solaris and strange
context class loader
Posted by Rainer Jung <ra...@kippdata.de>.
On 07.05.2010 14:23, Konstantin Kolinko wrote:
> 2010/5/7 Rainer Jung<ra...@kippdata.de>:
>> On 07.05.2010 11:00, Konstantin Kolinko wrote:
>>>
>>> 2010/5/7 Rainer Jung<ra...@kippdata.de>:
>>>>
>>>> I'm wondering why the PCKS Token
>>>> Poller thread was captured by the leak prevention. Since we know the
>>>> code,
>>>> it was because its context class loader was equal to the
>>>> WebappClassLoader
>>>> of /manager. That's what I don't understand. See my original post.
>>>>
>>>
>>> TCCL is inherited from the parent thread. It means that that thread
>>> was started either during request processing, or during webapplication
>>> startup.
>>
>> But unfortunately that doesn't provide consistent results.
>>
>> If I sart Tomcat without any webapp (but with ROOT context) the thread is
>> started too. The heap dump shows the system cl as the tccl, but during
>> restart or shutdown the leak prevention retrieves the tccl and finds it s
>> equal to the WebappClassLoader of the manager.
>>
>> I'm confused.
>>
>>> BTW, 4th result in
>>> http://www.google.com/search?q=SunPKCS11+TokenPoller
>>> gives source code for that class.
>>
>> I'll see, but I guess the point is more about who starts it when and why,
>> less about what it does.
>>
>
> What is this all about, I means this PKCS11 tokens?
> Is it something that is used in your configuration?
No, not at all. There's not HTTPS connector, no keystore no nothing.
The PKCS11 provider comes with the JDK but I'm not aware of anything
that uses it. Default Tomcat configuration plus log4j.
It's there after JDK start with some version of the JDK. It's not there
if I only start a plain Java test doing only a sleep using the same JDK.
> I mean, what are those "tokens" that it works with? (for a newbie)
> Do you use JAAS?
Not that I'm aware of.
> From the code, e.g.
> SunPKCS11#uninitToken() calls
> 1) destroyPoller()
> 2) createPoller()
>
> which restarts the thread. So the one at shutdown time might be
> different from the one that you see when Tomcat starts.
Hmmm. Will dig deeper over the weekend.
Regards,
Rainer
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: clearReferencesThreads, Poller SunPKCS11-Solaris and strange
context class loader
Posted by Konstantin Kolinko <kn...@gmail.com>.
2010/5/7 Rainer Jung <ra...@kippdata.de>:
> On 07.05.2010 11:00, Konstantin Kolinko wrote:
>>
>> 2010/5/7 Rainer Jung<ra...@kippdata.de>:
>>>
>>> I'm wondering why the PCKS Token
>>> Poller thread was captured by the leak prevention. Since we know the
>>> code,
>>> it was because its context class loader was equal to the
>>> WebappClassLoader
>>> of /manager. That's what I don't understand. See my original post.
>>>
>>
>> TCCL is inherited from the parent thread. It means that that thread
>> was started either during request processing, or during webapplication
>> startup.
>
> But unfortunately that doesn't provide consistent results.
>
> If I sart Tomcat without any webapp (but with ROOT context) the thread is
> started too. The heap dump shows the system cl as the tccl, but during
> restart or shutdown the leak prevention retrieves the tccl and finds it s
> equal to the WebappClassLoader of the manager.
>
> I'm confused.
>
>> BTW, 4th result in
>> http://www.google.com/search?q=SunPKCS11+TokenPoller
>> gives source code for that class.
>
> I'll see, but I guess the point is more about who starts it when and why,
> less about what it does.
>
What is this all about, I means this PKCS11 tokens?
Is it something that is used in your configuration?
I mean, what are those "tokens" that it works with? (for a newbie)
Do you use JAAS?
>From the code, e.g.
SunPKCS11#uninitToken() calls
1) destroyPoller()
2) createPoller()
which restarts the thread. So the one at shutdown time might be
different from the one that you see when Tomcat starts.
Best regards,
Konstantin Kolinko
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: clearReferencesThreads, Poller SunPKCS11-Solaris and strange
context class loader
Posted by Rainer Jung <ra...@kippdata.de>.
On 07.05.2010 11:00, Konstantin Kolinko wrote:
> 2010/5/7 Rainer Jung<ra...@kippdata.de>:
>> I'm wondering why the PCKS Token
>> Poller thread was captured by the leak prevention. Since we know the code,
>> it was because its context class loader was equal to the WebappClassLoader
>> of /manager. That's what I don't understand. See my original post.
>>
>
> TCCL is inherited from the parent thread. It means that that thread
> was started either during request processing, or during webapplication
> startup.
But unfortunately that doesn't provide consistent results.
If I sart Tomcat without any webapp (but with ROOT context) the thread
is started too. The heap dump shows the system cl as the tccl, but
during restart or shutdown the leak prevention retrieves the tccl and
finds it s equal to the WebappClassLoader of the manager.
I'm confused.
> BTW, 4th result in
> http://www.google.com/search?q=SunPKCS11+TokenPoller
> gives source code for that class.
I'll see, but I guess the point is more about who starts it when and
why, less about what it does.
Regards,
Rainer
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: clearReferencesThreads, Poller SunPKCS11-Solaris and strange
context class loader
Posted by Konstantin Kolinko <kn...@gmail.com>.
2010/5/7 Rainer Jung <ra...@kippdata.de>:
> I'm wondering why the PCKS Token
> Poller thread was captured by the leak prevention. Since we know the code,
> it was because its context class loader was equal to the WebappClassLoader
> of /manager. That's what I don't understand. See my original post.
>
TCCL is inherited from the parent thread. It means that that thread
was started either during request processing, or during webapplication
startup.
BTW, 4th result in
http://www.google.com/search?q=SunPKCS11+TokenPoller
gives source code for that class.
Best regards,
Konstantin Kolinko
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: clearReferencesThreads, Poller SunPKCS11-Solaris and strange
context class loader
Posted by Rainer Jung <ra...@kippdata.de>.
On 06.05.2010 23:24, Sylvain Laurent wrote:
> When you analyzed the heap dump, what path to GC roots was retaining the classloader in memory ?
I did the heap dump before the context restart.
Note: I don't want to debug a memory leak. I'm wondering why the PCKS
Token Poller thread was captured by the leak prevention. Since we know
the code, it was because its context class loader was equal to the
WebappClassLoader of /manager. That's what I don't understand. See my
original post.
Regards,
Rainer
> On 6 mai 2010, at 20:51, Rainer Jung wrote:
>
>> While doing some testing with 6.0.26 I noticed, that when shutting it down it logs an error about thread "Poller SunPKCS11-Solaris" not being stopped. When I add the more explicit logging from tc6 trunk, it says the thread was started by /manager.
>>
>> The thread sits in
>>
>> java.lang.Thread.State: TIMED_WAITING (sleeping)
>> at java.lang.Thread.sleep(Native Method)
>> at sun.security.pkcs11.SunPKCS11$TokenPoller.run(SunPKCS11.java:681)
>> at java.lang.Thread.run(Thread.java:619)
>>
>> and is started for some version of the JDK (noted on Solaris for e.g. 1.5.0_22 and 1.6.0_3, not for 1.6.0_17) even without having anything related to keystores, https or similar configured in Tomcat (default config). The only non-default is using Log4J instead of Juli.
>>
>> What is strange, is that the check whether the context class loader of the thread is equal to the WebappClassLoader of the context to unload passes. I added an additional output by explicitely printing the contextName of the two loaders and in fact both print "/manager".
>>
>> When deploying two instances of the manager and reloading the original instance with the additional one, I get the same warning.
>>
>> In a heap dump, it seems the context class loader of the thread is the system class loader and not of type WebappClassLoader. Any idea, why the context class loader test fails? Is there any reason why the context class loader of the thread might change when doing the manager reload or shutdown?
>>
>> Regards,
>>
>> Rainer
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: clearReferencesThreads, Poller SunPKCS11-Solaris and strange context class loader
Posted by Sylvain Laurent <sy...@m4x.org>.
When you analyzed the heap dump, what path to GC roots was retaining the classloader in memory ?
On 6 mai 2010, at 20:51, Rainer Jung wrote:
> While doing some testing with 6.0.26 I noticed, that when shutting it down it logs an error about thread "Poller SunPKCS11-Solaris" not being stopped. When I add the more explicit logging from tc6 trunk, it says the thread was started by /manager.
>
> The thread sits in
>
> java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at sun.security.pkcs11.SunPKCS11$TokenPoller.run(SunPKCS11.java:681)
> at java.lang.Thread.run(Thread.java:619)
>
> and is started for some version of the JDK (noted on Solaris for e.g. 1.5.0_22 and 1.6.0_3, not for 1.6.0_17) even without having anything related to keystores, https or similar configured in Tomcat (default config). The only non-default is using Log4J instead of Juli.
>
> What is strange, is that the check whether the context class loader of the thread is equal to the WebappClassLoader of the context to unload passes. I added an additional output by explicitely printing the contextName of the two loaders and in fact both print "/manager".
>
> When deploying two instances of the manager and reloading the original instance with the additional one, I get the same warning.
>
> In a heap dump, it seems the context class loader of the thread is the system class loader and not of type WebappClassLoader. Any idea, why the context class loader test fails? Is there any reason why the context class loader of the thread might change when doing the manager reload or shutdown?
>
> Regards,
>
> Rainer
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: dev-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[Resolved] clearReferencesThreads, Poller SunPKCS11-Solaris and strange
context class loader
Posted by Rainer Jung <ra...@kippdata.de>.
On 06.05.2010 20:51, Rainer Jung wrote:
> While doing some testing with 6.0.26 I noticed, that when shutting it
> down it logs an error about thread "Poller SunPKCS11-Solaris" not being
> stopped. When I add the more explicit logging from tc6 trunk, it says
> the thread was started by /manager.
>
> The thread sits in
>
> java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at sun.security.pkcs11.SunPKCS11$TokenPoller.run(SunPKCS11.java:681)
> at java.lang.Thread.run(Thread.java:619)
>
> and is started for some version of the JDK (noted on Solaris for e.g.
> 1.5.0_22 and 1.6.0_3, not for 1.6.0_17) even without having anything
> related to keystores, https or similar configured in Tomcat (default
> config). The only non-default is using Log4J instead of Juli.
>
> What is strange, is that the check whether the context class loader of
> the thread is equal to the WebappClassLoader of the context to unload
> passes. I added an additional output by explicitely printing the
> contextName of the two loaders and in fact both print "/manager".
>
> When deploying two instances of the manager and reloading the original
> instance with the additional one, I get the same warning.
>
> In a heap dump, it seems the context class loader of the thread is the
> system class loader and not of type WebappClassLoader. Any idea, why the
> context class loader test fails? Is there any reason why the context
> class loader of the thread might change when doing the manager reload or
> shutdown?
1) Why is the class loader in the heap dump not the WebappClassLoader?
For producing a heap dump, I added
"-Dcom.sun.management.jmxremote=true". As a side effect java.security
got initialized much earlier and before Tomcat startup happened. So the
poller thread was started earlier and had the system class loader as
TCCL. I noticed that by adding "-verbose:class" to track class loading.
2) What triggers startup of that thread in normal Tomcat?
When *not* using the jmx system properties, the thread starts during
webapp deployment. StandardManager inside start() initializes random
numbers by calling generateSessionId(), and that method in ManagerBase
calls getDigest(), which calls java.security.MessageDigest.getInstance()
which finally initializes the Java Security Providers and starts the
poller thread.
I didn't actually check it, but I expect the TCCL of the main thread
during that stage of webapp deployment is the webapp class loader.
If I don't deploy the manager webapp, then the poller is simply started
during the ROOT webapp deployment. Whatever context is deployed first
starts the thread.
3) Why isn't it there for all Java versions?
Sun applied a fix to the SunPKCS11 class for the so-called soft token.
The thread is now started only, if the token is "removable" and the soft
token is not. The patch is not part e.g. of old Java 6 versions.
So for older JDK on Solaris 10, there is always a so-called soft token,
and initializing the Java security providers will launch the thread. For
older Solaris (e.g. 8) there is no soft token and thus no thread. For
newer JDK patch versions, the soft token is identified to not be
removable and again no poller thread is started.
4) Do we have to fix anything?
If users do not actually use a removable token, upgrading the JDK is
enough. Once users start to complain about the leak warning when using a
removable token, it would suffice to initialize the Java security
providers before deploying the webapps, so that the thread has the
system or catalina loader as TCCL. This can be done e.g. by calling
java.security.Security.getProviders(). But actually I'm not sure,
whether the warning is a false positive. It's quite possible, that the
poller thread actually does imply a perm gen leak.
Thanks to all who provided valuable pointers.
Regards,
Rainer
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org