You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@tomcat.apache.org by bu...@apache.org on 2021/09/06 07:53:40 UTC

[Bug 65553] New: Orphaned thread by JNDIRealm / clearReferencesThreads reports memory leak

https://bz.apache.org/bugzilla/show_bug.cgi?id=65553

Bug ID: 65553
Summary: Orphaned thread by JNDIRealm / clearReferencesThreads
reports memory leak
Product: Tomcat 9
Version: 9.0.52
Hardware: PC
Status: NEW
Severity: minor
Priority: P2
Component: Catalina
Assignee: dev@tomcat.apache.org
Reporter: Thomas.Hoffmann@speed4trade.com
Target Milestone: -----

Created attachment 38020
--> https://bz.apache.org/bugzilla/attachment.cgi?id=38020&action=edit
Stacktrace on creation of orphaned thread

Hello,

we are using the org.apache.catalina.realm.JNDIRealm for authentication of
users against our windows AD.
When undeploying the application, we see the following warning in our logs:

WARNING [Catalina-utility-1] org.apache.catalina.loader.Webapp
ClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to
have started a thread named [Thread-106] but has failed to stop it. This is
very likely to create a memory leak. Stack trace of thread:
java.base@11.0.3/java.net.SocketInputStream.socketRead0(Native Method)

java.base@11.0.3/java.net.SocketInputStream.socketRead(SocketInputStream.java:115)
java.base@11.0.3/java.net.SocketInputStream.read(SocketInputStream.java:168)
java.base@11.0.3/java.net.SocketInputStream.read(SocketInputStream.java:140)

java.base@11.0.3/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:448)

java.base@11.0.3/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:68)

java.base@11.0.3/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1104)

java.base@11.0.3/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:823)

java.base@11.0.3/java.io.BufferedInputStream.fill(BufferedInputStream.java:252)

java.base@11.0.3/java.io.BufferedInputStream.read1(BufferedInputStream.java:292)

java.base@11.0.3/java.io.BufferedInputStream.read(BufferedInputStream.java:351)
java.naming@11.0.3/com.sun.jndi.ldap.Connection.run(Connection.java:832)
java.base@11.0.3/java.lang.Thread.run(Thread.java:834)

The warning is not always shown but quite often.

Summary of the analysis of the problem:
On tomcat startup, the worker-thread is running under the tomcat classloader.
But when a reconnect happens, the thread is running with the classloader of
the web application and gets thus reported.

The details:
Digging into the problem via remote debugging showed the reason how this
happens:
During startup, Tomcat is initializing the JNDIRealm. The open-method of
JNDIRealm is switching the classloader between bootstrap-CL and tomcat-lib-CL,
depending on the attribute "useContextClassLoader".
Afterwards the context-Object is created (createDirContext). Within this
LdapCtx, an LdapClient is used to communicate with the AD-Server.
This LdapClient uses a com.sun.jndi.ldap.Connection for TCP communication.
This connection opens the reported Worker-Thread.
This can be seen at
https://github.com/AdoptOpenJDK/openjdk-jdk11/blob/master/src/java.naming/share/classes/com/sun/jndi/ldap/Connection.java
around line 243 --> worker = Obj.helper.createThread(this);

So far, so good.

Somehow, the com.sun.jndi.ldap.Connection is sometimes closed and the thread
dies. At least, the thread is not visible any more. Maybe because of a timeout
on the AD-server side or something else happened.
If a new user accesses the site, the JNDIRealm is authenticating the user.
This triggers the following chain (path is shortened):
JNDIRealm.getUserBySearch --> LdapCtx.dosearch --> LdapCtx.ensureOpen -->
LdapCtx.connect --> LdapClient.getInstance --> Connection.<init>

A detailed stack is attached as an image.

This call chain creates a new com.sun.jndi.ldap.Connection and thus a new
thread. But this time, the thread is connected to the classloader of the
web-application.
On undeployment, the thread is thus reported to be orphaned.

It was tested with Tomcat 9.0.52, Windows 10, OpenJDK 11.0.12_7.

As the authentication is conducted within tomcat, before the application is
triggered, thus the application can't handle that error.

The flag "useContextClassLoader" is not taken into account during
authentication.
During this process a re-connect can occur and create the thread with the wrong
classloader.

Thanks in advance,
Thomas

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org