You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by "Thomas Hoffmann (Speed4Trade GmbH)" <Th...@speed4trade.com.INVALID> on 2021/09/02 07:33:37 UTC
Orphaned thread by JNDIRealm / clearReferencesThreads reports memory
leak
Hello,
we are using the org.apache.catalina.realm.JNDIRealm for authentication of users against our windows AD.
When undeploying the application, we see the following warning in our logs:
WARNING [Catalina-utility-1] org.apache.catalina.loader.Webapp
ClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [Thread-106] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11.0.3/java.net.SocketInputStream.socketRead0(Native Method)
java.base@11.0.3/java.net.SocketInputStream.socketRead(SocketInputStream.java:115)
java.base@11.0.3/java.net.SocketInputStream.read(SocketInputStream.java:168)
java.base@11.0.3/java.net.SocketInputStream.read(SocketInputStream.java:140)
java.base@11.0.3/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:448)
java.base@11.0.3/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:68)
java.base@11.0.3/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1104)
java.base@11.0.3/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:823)
java.base@11.0.3/java.io.BufferedInputStream.fill(BufferedInputStream.java:252)
java.base@11.0.3/java.io.BufferedInputStream.read1(BufferedInputStream.java:292)
java.base@11.0.3/java.io.BufferedInputStream.read(BufferedInputStream.java:351)
java.naming@11.0.3/com.sun.jndi.ldap.Connection.run(Connection.java:832)
java.base@11.0.3/java.lang.Thread.run(Thread.java:834)
The warning is not always shown but quite often.
Summary of the analysis of the problem:
On tomcat startup, the worker-thread is running under the tomcat classloader. But when a reconnect happens, the thread is running with the classloader of the web application and gets thus reported.
The details:
Digging into the problem via remote debugging showed the reason how this happens:
During startup, Tomcat is initializing the JNDIRealm. The open-method of JNDIRealm is switching the classloader between bootstrap-CL and tomcat-lib-CL, depending on the attribute "useContextClassLoader".
Afterwards the context-Object is created (createDirContext). Within this LdapCtx, an LdapClient is used to communicate with the AD-Server.
This LdapClient uses a com.sun.jndi.ldap.Connection for TCP communication. This connection opens the reported Worker-Thread.
This can be seen at https://github.com/AdoptOpenJDK/openjdk-jdk11/blob/master/src/java.naming/share/classes/com/sun/jndi/ldap/Connection.java around line 243 --> worker = Obj.helper.createThread(this);
So far, so good.
Somehow, the com.sun.jndi.ldap.Connection is sometimes closed and the thread dies. At least, the thread is not visible any more. Maybe because of a timeout on the AD-server side or something else happened.
If a new user accesses the site, the JNDIRealm is authenticating the user.
This triggers the following chain (path is shortened): JNDIRealm.getUserBySearch --> LdapCtx.dosearch --> LdapCtx.ensureOpen --> LdapCtx.connect --> LdapClient.getInstance --> Connection.<init>
This creates a new com.sun.jndi.ldap.Connection and thus a new thread. But this time, the thread is connected to the classloader of the web-application.
On undeployment, the thread is thus reported to be orphaned.
It was tested with Tomcat 9.0.52, Windows 10, OpenJDK 11.0.12_7.
As the authentication is conducted within tomcat, before the application is triggered, I am not sure if the problem can be tackled on application side.
Thanks in advance,
Thomas
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: AW: AW: Orphaned thread by JNDIRealm / clearReferencesThreads
reports memory leak
Posted by Mark Thomas <ma...@apache.org>.
On 06/09/2021 08:47, Thomas Hoffmann (Speed4Trade GmbH) wrote:
> Hello Mark,
>
> thank you for taking a look at it and confirming the behaviour.
> I will file a bug as suggested.
Thanks, I see it.
I have added my own comment. I'll give others some time to comment and
aim to fix this for the October release round.
Mark
>
> Thank you and have a good start into the new week!
> Thomas
>
> -----Ursprüngliche Nachricht-----
> Von: Mark Thomas <ma...@apache.org>
> Gesendet: Montag, 6. September 2021 09:36
> An: users@tomcat.apache.org
> Betreff: Re: AW: Orphaned thread by JNDIRealm / clearReferencesThreads reports memory leak
>
> Hi Thomas.
>
> Got it.
>
> I think it would be best if you opened a Bugzilla entry for this.
>
> One view is that the JRE should not be doing this - or at least doing it in a way that doesn't run the risk of memory leaks in a Java EE / Jakarta EE environment. We have raised JRE bugs for issues like this in the past and they have been fixed.
>
> The counter view is that this thread is not a one-off and is not short-lived (the typical cases for JRE bugs that have been fixed) and Tomcat has already acknowledged - with the useContextClassLoader flag - that it needs to be taking action to ensure that any threads are created with the expected class loader.
>
> If the consensus is that Tomcat needs to handle this then we'd need to ensure that every version of authenticate() also set the class loader if required. We'd probably want to refactor the existing handling to make sure we don't try setting the class loader twice.
>
> Whatever changes we make we need to be able to reference them back to the discussion of why we made them and a Bugzilla issue is how we do that.
>
> Thanks,
>
> Mark
>
>
> On 05/09/2021 22:03, Thomas Hoffmann (Speed4Trade GmbH) wrote:
>> Hello Mark,
>>
>> thanks for your reply.
>> I already tried that option. But this flag only controls, how the InitialDirContext is created.
>> The worker thread within com.sun.jndi.ldap.Connection is created with that corresponding classloader the first time.
>>
>> The problem is when the worker-thread within the com.sun.jndi.ldap.Connection dies and is re-established again.
>> In that case, the flag useContextClassLoader is not considered any more because the InitialDirContext is already instantiated.
>>
>> The call stack when the InitialDirContext is already instantiated but the connection gets re-established looks like:
>> JNDIRealm.authenticate --> JNDIRealm.getUserBySearch -->
>> LdapCtx.dosearch --> LdapCtx.ensureOpen --> LdapCtx.connect -->
>> LdapClient.getInstance --> Connection.<init>
>>
>> In this call chain, the flag useContextClassLoader is not used any more as the InitialDirContext already exists.
>> The get() method just provides the existing JNDIConnection without switching any classloader.
>> Now however, the context classloader of the application is used, independent of the setting "useContextClassLoader".
>> Therefore, the second time when the worker thread gets created, it inherits the classloader of the application and is reported as leaking during undeployment.
>>
>> Greetings, Thomas
>>
>>
>>
>>
>> Von: Mark Thomas <ma...@apache.org>
>> Gesendet: Sonntag, 5. September 2021 11:55
>> An: users@tomcat.apache.org
>> Betreff: Re: Orphaned thread by JNDIRealm / clearReferencesThreads
>> reports memory leak
>>
>> Thomas,
>>
>> Try setting:
>>
>> useContextClassLoader="false"
>>
>> for the JNDIRealm.
>>
>> Mark
>>
>>
>> On 02/09/2021 08:33, Thomas Hoffmann (Speed4Trade GmbH) wrote:
>>> Hello,
>>>
>>> we are using the org.apache.catalina.realm.JNDIRealm for authentication of users against our windows AD.
>>> When undeploying the application, we see the following warning in our logs:
>>>
>>> WARNING [Catalina-utility-1] org.apache.catalina.loader.Webapp
>>> ClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [Thread-106] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
>>> java.base@11.0.3/java.net.SocketInputStream.socketRead0(Native
>>> Method)
>>>
>>> java.base@11.0.3/java.net.SocketInputStream.socketRead(SocketInputStr
>>> eam.java:115)
>>>
>>> java.base@11.0.3/java.net.SocketInputStream.read(SocketInputStream.ja
>>> va:168)
>>>
>>> java.base@11.0.3/java.net.SocketInputStream.read(SocketInputStream.ja
>>> va:140)
>>>
>>> java.base@11.0.3/sun.security.ssl.SSLSocketInputRecord.read(SSLSocket
>>> InputRecord.java:448)
>>>
>>> java.base@11.0.3/sun.security.ssl.SSLSocketInputRecord.bytesInComplet
>>> ePacket(SSLSocketInputRecord.java:68)
>>>
>>> java.base@11.0.3/sun.security.ssl.SSLSocketImpl.readApplicationRecord
>>> (SSLSocketImpl.java:1104)
>>>
>>> java.base@11.0.3/sun.security.ssl.SSLSocketImpl$AppInputStream.read(S
>>> SLSocketImpl.java:823)
>>>
>>> java.base@11.0.3/java.io.BufferedInputStream.fill(BufferedInputStream
>>> .java:252)
>>>
>>> java.base@11.0.3/java.io.BufferedInputStream.read1(BufferedInputStrea
>>> m.java:292)
>>>
>>> java.base@11.0.3/java.io.BufferedInputStream.read(BufferedInputStream
>>> .java:351)
>>>
>>> java.naming@11.0.3/com.sun.jndi.ldap.Connection.run(Connection.java:8
>>> 32)
>>> java.base@11.0.3/java.lang.Thread.run(Thread.java:834)
>>>
>>> The warning is not always shown but quite often.
>>>
>>> Summary of the analysis of the problem:
>>> On tomcat startup, the worker-thread is running under the tomcat classloader. But when a reconnect happens, the thread is running with the classloader of the web application and gets thus reported.
>>>
>>> The details:
>>> Digging into the problem via remote debugging showed the reason how this happens:
>>> During startup, Tomcat is initializing the JNDIRealm. The open-method of JNDIRealm is switching the classloader between bootstrap-CL and tomcat-lib-CL, depending on the attribute "useContextClassLoader".
>>> Afterwards the context-Object is created (createDirContext). Within this LdapCtx, an LdapClient is used to communicate with the AD-Server.
>>> This LdapClient uses a com.sun.jndi.ldap.Connection for TCP communication. This connection opens the reported Worker-Thread.
>>> This can be seen at
>>> https://github.com/AdoptOpenJDK/openjdk-jdk11/blob/master/src/java.na
>>> ming/share/classes/com/sun/jndi/ldap/Connection.java around line 243
>>> --> worker = Obj.helper.createThread(this);
>>>
>>> So far, so good.
>>>
>>> Somehow, the com.sun.jndi.ldap.Connection is sometimes closed and the thread dies. At least, the thread is not visible any more. Maybe because of a timeout on the AD-server side or something else happened.
>>> If a new user accesses the site, the JNDIRealm is authenticating the user.
>>> This triggers the following chain (path is shortened):
>>> JNDIRealm.getUserBySearch --> LdapCtx.dosearch --> LdapCtx.ensureOpen --> LdapCtx.connect --> LdapClient.getInstance --> Connection.<init> This creates a new com.sun.jndi.ldap.Connection and thus a new thread. But this time, the thread is connected to the classloader of the web-application.
>>> On undeployment, the thread is thus reported to be orphaned.
>>>
>>> It was tested with Tomcat 9.0.52, Windows 10, OpenJDK 11.0.12_7.
>>>
>>> As the authentication is conducted within tomcat, before the application is triggered, I am not sure if the problem can be tackled on application side.
>>>
>>> Thanks in advance,
>>> Thomas
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
AW: AW: Orphaned thread by JNDIRealm / clearReferencesThreads reports
memory leak
Posted by "Thomas Hoffmann (Speed4Trade GmbH)" <Th...@speed4trade.com.INVALID>.
Hello Mark,
thank you for taking a look at it and confirming the behaviour.
I will file a bug as suggested.
Thank you and have a good start into the new week!
Thomas
-----Ursprüngliche Nachricht-----
Von: Mark Thomas <ma...@apache.org>
Gesendet: Montag, 6. September 2021 09:36
An: users@tomcat.apache.org
Betreff: Re: AW: Orphaned thread by JNDIRealm / clearReferencesThreads reports memory leak
Hi Thomas.
Got it.
I think it would be best if you opened a Bugzilla entry for this.
One view is that the JRE should not be doing this - or at least doing it in a way that doesn't run the risk of memory leaks in a Java EE / Jakarta EE environment. We have raised JRE bugs for issues like this in the past and they have been fixed.
The counter view is that this thread is not a one-off and is not short-lived (the typical cases for JRE bugs that have been fixed) and Tomcat has already acknowledged - with the useContextClassLoader flag - that it needs to be taking action to ensure that any threads are created with the expected class loader.
If the consensus is that Tomcat needs to handle this then we'd need to ensure that every version of authenticate() also set the class loader if required. We'd probably want to refactor the existing handling to make sure we don't try setting the class loader twice.
Whatever changes we make we need to be able to reference them back to the discussion of why we made them and a Bugzilla issue is how we do that.
Thanks,
Mark
On 05/09/2021 22:03, Thomas Hoffmann (Speed4Trade GmbH) wrote:
> Hello Mark,
>
> thanks for your reply.
> I already tried that option. But this flag only controls, how the InitialDirContext is created.
> The worker thread within com.sun.jndi.ldap.Connection is created with that corresponding classloader the first time.
>
> The problem is when the worker-thread within the com.sun.jndi.ldap.Connection dies and is re-established again.
> In that case, the flag useContextClassLoader is not considered any more because the InitialDirContext is already instantiated.
>
> The call stack when the InitialDirContext is already instantiated but the connection gets re-established looks like:
> JNDIRealm.authenticate --> JNDIRealm.getUserBySearch -->
> LdapCtx.dosearch --> LdapCtx.ensureOpen --> LdapCtx.connect -->
> LdapClient.getInstance --> Connection.<init>
>
> In this call chain, the flag useContextClassLoader is not used any more as the InitialDirContext already exists.
> The get() method just provides the existing JNDIConnection without switching any classloader.
> Now however, the context classloader of the application is used, independent of the setting "useContextClassLoader".
> Therefore, the second time when the worker thread gets created, it inherits the classloader of the application and is reported as leaking during undeployment.
>
> Greetings, Thomas
>
>
>
>
> Von: Mark Thomas <ma...@apache.org>
> Gesendet: Sonntag, 5. September 2021 11:55
> An: users@tomcat.apache.org
> Betreff: Re: Orphaned thread by JNDIRealm / clearReferencesThreads
> reports memory leak
>
> Thomas,
>
> Try setting:
>
> useContextClassLoader="false"
>
> for the JNDIRealm.
>
> Mark
>
>
> On 02/09/2021 08:33, Thomas Hoffmann (Speed4Trade GmbH) wrote:
>> Hello,
>>
>> we are using the org.apache.catalina.realm.JNDIRealm for authentication of users against our windows AD.
>> When undeploying the application, we see the following warning in our logs:
>>
>> WARNING [Catalina-utility-1] org.apache.catalina.loader.Webapp
>> ClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [Thread-106] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
>> java.base@11.0.3/java.net.SocketInputStream.socketRead0(Native
>> Method)
>>
>> java.base@11.0.3/java.net.SocketInputStream.socketRead(SocketInputStr
>> eam.java:115)
>>
>> java.base@11.0.3/java.net.SocketInputStream.read(SocketInputStream.ja
>> va:168)
>>
>> java.base@11.0.3/java.net.SocketInputStream.read(SocketInputStream.ja
>> va:140)
>>
>> java.base@11.0.3/sun.security.ssl.SSLSocketInputRecord.read(SSLSocket
>> InputRecord.java:448)
>>
>> java.base@11.0.3/sun.security.ssl.SSLSocketInputRecord.bytesInComplet
>> ePacket(SSLSocketInputRecord.java:68)
>>
>> java.base@11.0.3/sun.security.ssl.SSLSocketImpl.readApplicationRecord
>> (SSLSocketImpl.java:1104)
>>
>> java.base@11.0.3/sun.security.ssl.SSLSocketImpl$AppInputStream.read(S
>> SLSocketImpl.java:823)
>>
>> java.base@11.0.3/java.io.BufferedInputStream.fill(BufferedInputStream
>> .java:252)
>>
>> java.base@11.0.3/java.io.BufferedInputStream.read1(BufferedInputStrea
>> m.java:292)
>>
>> java.base@11.0.3/java.io.BufferedInputStream.read(BufferedInputStream
>> .java:351)
>>
>> java.naming@11.0.3/com.sun.jndi.ldap.Connection.run(Connection.java:8
>> 32)
>> java.base@11.0.3/java.lang.Thread.run(Thread.java:834)
>>
>> The warning is not always shown but quite often.
>>
>> Summary of the analysis of the problem:
>> On tomcat startup, the worker-thread is running under the tomcat classloader. But when a reconnect happens, the thread is running with the classloader of the web application and gets thus reported.
>>
>> The details:
>> Digging into the problem via remote debugging showed the reason how this happens:
>> During startup, Tomcat is initializing the JNDIRealm. The open-method of JNDIRealm is switching the classloader between bootstrap-CL and tomcat-lib-CL, depending on the attribute "useContextClassLoader".
>> Afterwards the context-Object is created (createDirContext). Within this LdapCtx, an LdapClient is used to communicate with the AD-Server.
>> This LdapClient uses a com.sun.jndi.ldap.Connection for TCP communication. This connection opens the reported Worker-Thread.
>> This can be seen at
>> https://github.com/AdoptOpenJDK/openjdk-jdk11/blob/master/src/java.na
>> ming/share/classes/com/sun/jndi/ldap/Connection.java around line 243
>> --> worker = Obj.helper.createThread(this);
>>
>> So far, so good.
>>
>> Somehow, the com.sun.jndi.ldap.Connection is sometimes closed and the thread dies. At least, the thread is not visible any more. Maybe because of a timeout on the AD-server side or something else happened.
>> If a new user accesses the site, the JNDIRealm is authenticating the user.
>> This triggers the following chain (path is shortened):
>> JNDIRealm.getUserBySearch --> LdapCtx.dosearch --> LdapCtx.ensureOpen --> LdapCtx.connect --> LdapClient.getInstance --> Connection.<init> This creates a new com.sun.jndi.ldap.Connection and thus a new thread. But this time, the thread is connected to the classloader of the web-application.
>> On undeployment, the thread is thus reported to be orphaned.
>>
>> It was tested with Tomcat 9.0.52, Windows 10, OpenJDK 11.0.12_7.
>>
>> As the authentication is conducted within tomcat, before the application is triggered, I am not sure if the problem can be tackled on application side.
>>
>> Thanks in advance,
>> Thomas
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: AW: Orphaned thread by JNDIRealm / clearReferencesThreads reports
memory leak
Posted by Mark Thomas <ma...@apache.org>.
Hi Thomas.
Got it.
I think it would be best if you opened a Bugzilla entry for this.
One view is that the JRE should not be doing this - or at least doing it
in a way that doesn't run the risk of memory leaks in a Java EE /
Jakarta EE environment. We have raised JRE bugs for issues like this in
the past and they have been fixed.
The counter view is that this thread is not a one-off and is not
short-lived (the typical cases for JRE bugs that have been fixed) and
Tomcat has already acknowledged - with the useContextClassLoader flag -
that it needs to be taking action to ensure that any threads are created
with the expected class loader.
If the consensus is that Tomcat needs to handle this then we'd need to
ensure that every version of authenticate() also set the class loader if
required. We'd probably want to refactor the existing handling to make
sure we don't try setting the class loader twice.
Whatever changes we make we need to be able to reference them back to
the discussion of why we made them and a Bugzilla issue is how we do that.
Thanks,
Mark
On 05/09/2021 22:03, Thomas Hoffmann (Speed4Trade GmbH) wrote:
> Hello Mark,
>
> thanks for your reply.
> I already tried that option. But this flag only controls, how the InitialDirContext is created.
> The worker thread within com.sun.jndi.ldap.Connection is created with that corresponding classloader the first time.
>
> The problem is when the worker-thread within the com.sun.jndi.ldap.Connection dies and is re-established again.
> In that case, the flag useContextClassLoader is not considered any more because the InitialDirContext is already instantiated.
>
> The call stack when the InitialDirContext is already instantiated but the connection gets re-established looks like:
> JNDIRealm.authenticate --> JNDIRealm.getUserBySearch --> LdapCtx.dosearch --> LdapCtx.ensureOpen --> LdapCtx.connect --> LdapClient.getInstance --> Connection.<init>
>
> In this call chain, the flag useContextClassLoader is not used any more as the InitialDirContext already exists.
> The get() method just provides the existing JNDIConnection without switching any classloader.
> Now however, the context classloader of the application is used, independent of the setting "useContextClassLoader".
> Therefore, the second time when the worker thread gets created, it inherits the classloader of the application and is reported as leaking during undeployment.
>
> Greetings, Thomas
>
>
>
>
> Von: Mark Thomas <ma...@apache.org>
> Gesendet: Sonntag, 5. September 2021 11:55
> An: users@tomcat.apache.org
> Betreff: Re: Orphaned thread by JNDIRealm / clearReferencesThreads reports memory leak
>
> Thomas,
>
> Try setting:
>
> useContextClassLoader="false"
>
> for the JNDIRealm.
>
> Mark
>
>
> On 02/09/2021 08:33, Thomas Hoffmann (Speed4Trade GmbH) wrote:
>> Hello,
>>
>> we are using the org.apache.catalina.realm.JNDIRealm for authentication of users against our windows AD.
>> When undeploying the application, we see the following warning in our logs:
>>
>> WARNING [Catalina-utility-1] org.apache.catalina.loader.Webapp
>> ClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [Thread-106] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
>> java.base@11.0.3/java.net.SocketInputStream.socketRead0(Native Method)
>> java.base@11.0.3/java.net.SocketInputStream.socketRead(SocketInputStream.java:115)
>> java.base@11.0.3/java.net.SocketInputStream.read(SocketInputStream.java:168)
>> java.base@11.0.3/java.net.SocketInputStream.read(SocketInputStream.java:140)
>> java.base@11.0.3/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:448)
>> java.base@11.0.3/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:68)
>> java.base@11.0.3/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1104)
>> java.base@11.0.3/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:823)
>> java.base@11.0.3/java.io.BufferedInputStream.fill(BufferedInputStream.java:252)
>> java.base@11.0.3/java.io.BufferedInputStream.read1(BufferedInputStream.java:292)
>> java.base@11.0.3/java.io.BufferedInputStream.read(BufferedInputStream.java:351)
>> java.naming@11.0.3/com.sun.jndi.ldap.Connection.run(Connection.java:832)
>> java.base@11.0.3/java.lang.Thread.run(Thread.java:834)
>>
>> The warning is not always shown but quite often.
>>
>> Summary of the analysis of the problem:
>> On tomcat startup, the worker-thread is running under the tomcat classloader. But when a reconnect happens, the thread is running with the classloader of the web application and gets thus reported.
>>
>> The details:
>> Digging into the problem via remote debugging showed the reason how this happens:
>> During startup, Tomcat is initializing the JNDIRealm. The open-method of JNDIRealm is switching the classloader between bootstrap-CL and tomcat-lib-CL, depending on the attribute "useContextClassLoader".
>> Afterwards the context-Object is created (createDirContext). Within this LdapCtx, an LdapClient is used to communicate with the AD-Server.
>> This LdapClient uses a com.sun.jndi.ldap.Connection for TCP communication. This connection opens the reported Worker-Thread.
>> This can be seen at https://github.com/AdoptOpenJDK/openjdk-jdk11/blob/master/src/java.naming/share/classes/com/sun/jndi/ldap/Connection.java around line 243 --> worker = Obj.helper.createThread(this);
>>
>> So far, so good.
>>
>> Somehow, the com.sun.jndi.ldap.Connection is sometimes closed and the thread dies. At least, the thread is not visible any more. Maybe because of a timeout on the AD-server side or something else happened.
>> If a new user accesses the site, the JNDIRealm is authenticating the user.
>> This triggers the following chain (path is shortened): JNDIRealm.getUserBySearch --> LdapCtx.dosearch --> LdapCtx.ensureOpen --> LdapCtx.connect --> LdapClient.getInstance --> Connection.<init>
>> This creates a new com.sun.jndi.ldap.Connection and thus a new thread. But this time, the thread is connected to the classloader of the web-application.
>> On undeployment, the thread is thus reported to be orphaned.
>>
>> It was tested with Tomcat 9.0.52, Windows 10, OpenJDK 11.0.12_7.
>>
>> As the authentication is conducted within tomcat, before the application is triggered, I am not sure if the problem can be tackled on application side.
>>
>> Thanks in advance,
>> Thomas
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
AW: Orphaned thread by JNDIRealm / clearReferencesThreads reports
memory leak
Posted by "Thomas Hoffmann (Speed4Trade GmbH)" <Th...@speed4trade.com.INVALID>.
Hello Mark,
thanks for your reply.
I already tried that option. But this flag only controls, how the InitialDirContext is created.
The worker thread within com.sun.jndi.ldap.Connection is created with that corresponding classloader the first time.
The problem is when the worker-thread within the com.sun.jndi.ldap.Connection dies and is re-established again.
In that case, the flag useContextClassLoader is not considered any more because the InitialDirContext is already instantiated.
The call stack when the InitialDirContext is already instantiated but the connection gets re-established looks like:
JNDIRealm.authenticate --> JNDIRealm.getUserBySearch --> LdapCtx.dosearch --> LdapCtx.ensureOpen --> LdapCtx.connect --> LdapClient.getInstance --> Connection.<init>
In this call chain, the flag useContextClassLoader is not used any more as the InitialDirContext already exists.
The get() method just provides the existing JNDIConnection without switching any classloader.
Now however, the context classloader of the application is used, independent of the setting "useContextClassLoader".
Therefore, the second time when the worker thread gets created, it inherits the classloader of the application and is reported as leaking during undeployment.
Greetings, Thomas
Von: Mark Thomas <ma...@apache.org>
Gesendet: Sonntag, 5. September 2021 11:55
An: users@tomcat.apache.org
Betreff: Re: Orphaned thread by JNDIRealm / clearReferencesThreads reports memory leak
Thomas,
Try setting:
useContextClassLoader="false"
for the JNDIRealm.
Mark
On 02/09/2021 08:33, Thomas Hoffmann (Speed4Trade GmbH) wrote:
> Hello,
>
> we are using the org.apache.catalina.realm.JNDIRealm for authentication of users against our windows AD.
> When undeploying the application, we see the following warning in our logs:
>
> WARNING [Catalina-utility-1] org.apache.catalina.loader.Webapp
> ClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [Thread-106] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
> java.base@11.0.3/java.net.SocketInputStream.socketRead0(Native Method)
> java.base@11.0.3/java.net.SocketInputStream.socketRead(SocketInputStream.java:115)
> java.base@11.0.3/java.net.SocketInputStream.read(SocketInputStream.java:168)
> java.base@11.0.3/java.net.SocketInputStream.read(SocketInputStream.java:140)
> java.base@11.0.3/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:448)
> java.base@11.0.3/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:68)
> java.base@11.0.3/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1104)
> java.base@11.0.3/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:823)
> java.base@11.0.3/java.io.BufferedInputStream.fill(BufferedInputStream.java:252)
> java.base@11.0.3/java.io.BufferedInputStream.read1(BufferedInputStream.java:292)
> java.base@11.0.3/java.io.BufferedInputStream.read(BufferedInputStream.java:351)
> java.naming@11.0.3/com.sun.jndi.ldap.Connection.run(Connection.java:832)
> java.base@11.0.3/java.lang.Thread.run(Thread.java:834)
>
> The warning is not always shown but quite often.
>
> Summary of the analysis of the problem:
> On tomcat startup, the worker-thread is running under the tomcat classloader. But when a reconnect happens, the thread is running with the classloader of the web application and gets thus reported.
>
> The details:
> Digging into the problem via remote debugging showed the reason how this happens:
> During startup, Tomcat is initializing the JNDIRealm. The open-method of JNDIRealm is switching the classloader between bootstrap-CL and tomcat-lib-CL, depending on the attribute "useContextClassLoader".
> Afterwards the context-Object is created (createDirContext). Within this LdapCtx, an LdapClient is used to communicate with the AD-Server.
> This LdapClient uses a com.sun.jndi.ldap.Connection for TCP communication. This connection opens the reported Worker-Thread.
> This can be seen at https://github.com/AdoptOpenJDK/openjdk-jdk11/blob/master/src/java.naming/share/classes/com/sun/jndi/ldap/Connection.java around line 243 --> worker = Obj.helper.createThread(this);
>
> So far, so good.
>
> Somehow, the com.sun.jndi.ldap.Connection is sometimes closed and the thread dies. At least, the thread is not visible any more. Maybe because of a timeout on the AD-server side or something else happened.
> If a new user accesses the site, the JNDIRealm is authenticating the user.
> This triggers the following chain (path is shortened): JNDIRealm.getUserBySearch --> LdapCtx.dosearch --> LdapCtx.ensureOpen --> LdapCtx.connect --> LdapClient.getInstance --> Connection.<init>
> This creates a new com.sun.jndi.ldap.Connection and thus a new thread. But this time, the thread is connected to the classloader of the web-application.
> On undeployment, the thread is thus reported to be orphaned.
>
> It was tested with Tomcat 9.0.52, Windows 10, OpenJDK 11.0.12_7.
>
> As the authentication is conducted within tomcat, before the application is triggered, I am not sure if the problem can be tackled on application side.
>
> Thanks in advance,
> Thomas
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Orphaned thread by JNDIRealm / clearReferencesThreads reports
memory leak
Posted by Mark Thomas <ma...@apache.org>.
Thomas,
Try setting:
useContextClassLoader="false"
for the JNDIRealm.
Mark
On 02/09/2021 08:33, Thomas Hoffmann (Speed4Trade GmbH) wrote:
> Hello,
>
> we are using the org.apache.catalina.realm.JNDIRealm for authentication of users against our windows AD.
> When undeploying the application, we see the following warning in our logs:
>
> WARNING [Catalina-utility-1] org.apache.catalina.loader.Webapp
> ClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [Thread-106] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
> java.base@11.0.3/java.net.SocketInputStream.socketRead0(Native Method)
> java.base@11.0.3/java.net.SocketInputStream.socketRead(SocketInputStream.java:115)
> java.base@11.0.3/java.net.SocketInputStream.read(SocketInputStream.java:168)
> java.base@11.0.3/java.net.SocketInputStream.read(SocketInputStream.java:140)
> java.base@11.0.3/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:448)
> java.base@11.0.3/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:68)
> java.base@11.0.3/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1104)
> java.base@11.0.3/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:823)
> java.base@11.0.3/java.io.BufferedInputStream.fill(BufferedInputStream.java:252)
> java.base@11.0.3/java.io.BufferedInputStream.read1(BufferedInputStream.java:292)
> java.base@11.0.3/java.io.BufferedInputStream.read(BufferedInputStream.java:351)
> java.naming@11.0.3/com.sun.jndi.ldap.Connection.run(Connection.java:832)
> java.base@11.0.3/java.lang.Thread.run(Thread.java:834)
>
> The warning is not always shown but quite often.
>
> Summary of the analysis of the problem:
> On tomcat startup, the worker-thread is running under the tomcat classloader. But when a reconnect happens, the thread is running with the classloader of the web application and gets thus reported.
>
> The details:
> Digging into the problem via remote debugging showed the reason how this happens:
> During startup, Tomcat is initializing the JNDIRealm. The open-method of JNDIRealm is switching the classloader between bootstrap-CL and tomcat-lib-CL, depending on the attribute "useContextClassLoader".
> Afterwards the context-Object is created (createDirContext). Within this LdapCtx, an LdapClient is used to communicate with the AD-Server.
> This LdapClient uses a com.sun.jndi.ldap.Connection for TCP communication. This connection opens the reported Worker-Thread.
> This can be seen at https://github.com/AdoptOpenJDK/openjdk-jdk11/blob/master/src/java.naming/share/classes/com/sun/jndi/ldap/Connection.java around line 243 --> worker = Obj.helper.createThread(this);
>
> So far, so good.
>
> Somehow, the com.sun.jndi.ldap.Connection is sometimes closed and the thread dies. At least, the thread is not visible any more. Maybe because of a timeout on the AD-server side or something else happened.
> If a new user accesses the site, the JNDIRealm is authenticating the user.
> This triggers the following chain (path is shortened): JNDIRealm.getUserBySearch --> LdapCtx.dosearch --> LdapCtx.ensureOpen --> LdapCtx.connect --> LdapClient.getInstance --> Connection.<init>
> This creates a new com.sun.jndi.ldap.Connection and thus a new thread. But this time, the thread is connected to the classloader of the web-application.
> On undeployment, the thread is thus reported to be orphaned.
>
> It was tested with Tomcat 9.0.52, Windows 10, OpenJDK 11.0.12_7.
>
> As the authentication is conducted within tomcat, before the application is triggered, I am not sure if the problem can be tackled on application side.
>
> Thanks in advance,
> Thomas
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org