[jira] [Comment Edited] (CAMEL-17712) Memory leak in DefaultCamelContext reported by Tomcat 10

["Thomas Hoffmann (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CAMEL-17712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17514929#comment-17514929 ] 

Thomas Hoffmann edited comment on CAMEL-17712 at 3/30/22, 8:39 PM:
-------------------------------------------------------------------

Hello,

I tried the new version 3.14.2 but the memory leak is not fully fixed unfortunately.

Therefore I digged deeper into the problem and figured out the problem which lies a bit deeper than I tought.

The problem is when using ThreadLocal variables within CAMEL in multithreaded environments like application servers. The last fix only works in single threaded environments unfortunately.

*Background:*

Each thread in Java has a member variable or map "threadlocals" which stores the values which are managed by the Helper-Class "ThreadLocal", see:

[https://github.com/AdoptOpenJDK/openjdk-jdk11/blob/master/src/java.base/share/classes/java/lang/Thread.java]

The Java-Class ThreadLocal therefore doesn't store any variable by itself but attaches every value to the current thread (within their threadlocals map).

*Reason for the memory leak:*

I explain the problem with the example of DefaultCamelContext:

The ThreadLocal variable was introduced there in version 3.9:

[https://github.com/apache/camel/blob/camel-3.9.x/core/camel-core-engine/src/main/java/org/apache/camel/impl/DefaultCamelContext.java]

When a multithreaded application server (e.g. Tomcat or glassfish) starts a CamelContext, the DefaultCamelContext.java stores via the ThreadLocal-class and member variable OPTIONS the values (OptionHolder) into the current worker thread. This worker thread might be e.g. the thread worker-1. Some http calls to the webserver might be processed with the thread worker-2. This thread gets it's own option values when working with the started CamelContext. This new options are again attached to the current thread, in this case attached to worker-2.
If the application is undeployed, this might be processed by worker-3 but worker-3 can't clean up the options of the DefaultCamelContext which are stored in other worker-threads. Therefore the worker-1 and worker-2 thread still hold references to the CAMEL objects and thus they won't get garbage collected.

*Further occurences:*

The same problem occurs in class DefaultReactiveExecutor.java.

When using ThreadLocal-classes, the worker-threads of the application server get "polluted" with Camel-classes and the instances stay in memory, even when the application is undeployed because the worker-thread of the applicatoin server are re-used for subsequent http requests across the hosted applications. Thus the worker threads hold references to Camel classes and can't be garbage collected.

*Possible solutions:*

I am not sure, why there is a need to use ThreadLocal class in the mentioned Camel classes because I see no reason that every (worker) thread should have their own set of camel options, camel workers etc. Maybe the ThreadLocal classes where just used to prevent multi-threaded issues?

I currently see two options to solve the problem:

1) Make sure the ThreadLocal values are cleaned up before the method exits (via remove() method)

2) Replace the ThreadLocal variable with something else, if a per-thread variable is not needed.

I hope I could explain the complex problem well enough. If further information is needed, just drop a line. 

Thank you in advance! Thomas

 

 


was (Author: tom_s4t):
Hello,

I tried the new version 3.14.2 but the memory leak is not fully fixed unfortunately.

Therefore I digged deeper into the problem and figured out the problem which lies a bit deeper than I tought.

The problem is when using ThreadLocal variables within CAMEL in multithreaded environments like application servers. The last fix only works in single threaded environments unfortunately.

*Background:*

Each thread in Java has a member variable or map "threadlocals" which stores the values which are managed by the Helper-Class "ThreadLocal", see:

[https://github.com/AdoptOpenJDK/openjdk-jdk11/blob/master/src/java.base/share/classes/java/lang/Thread.java]

The Java-Class ThreadLocal therefore doesn't store any variable by itself but attaches every value to the current thread (within their threadlocals map).

*Reason for the memory leak:*

I explain the problem with the example of DefaultCamelContext:

The ThreadLocal variable was introduced there in version 3.9:

[https://github.com/apache/camel/blob/camel-3.9.x/core/camel-core-engine/src/main/java/org/apache/camel/impl/DefaultCamelContext.java]

When a multithreaded application server (e.g. Tomcat or glassfish) starts a CamelContext, the DefaultCamelContext.java stores via the ThreadLocal-class and member variable OPTIONS the values (OptionHolder) into the current worker thread. This worker thread might be e.g. the thread worker-1. Some http calls to the webserver might be processed with the thread worker-2. This thread gets it's own option values when working with the started CamelContext. This new options are again attached to the current thread, in this case attached to worker-2.
If the application is undeployed, this might be processed by worker-3 but worker-3 can't clean up the options of the DefaultCamelContext which are stored in other worker-threads. Therefore the worker-1 and worker-2 thread still hold references to the CAMEL objects and thus they won't get garbage collected.

*Further occurences:*

The same problem occurs in other classes, like DefaultReactiveExecutor.java and AbstractCamelContext.java.

When using ThreadLocal-classes, the worker-threads of the application server get "polluted" with Camel-classes and the instances stay in memory, even when the application is undeployed because the worker-thread of the applicatoin server are re-used for subsequent http requests across the hosted applications. Thus the worker threads hold references to Camel classes and can't be garbage collected.

*Possible solutions:*

I am not sure, why there is a need to use ThreadLocal class in the mentioned Camel classes because I see no reason that every (worker) thread should have their own set of camel options, camel workers etc. Maybe the ThreadLocal classes where just used to prevent multi-threaded issues?

I currently see two options to solve the problem:

1) Make sure the ThreadLocal values are cleaned up before the method exits (via remove() method)

2) Replace the ThreadLocal variable with something else, if a per-thread variable is not needed.

I hope I could explain the complex problem well enough. If further information is needed, just drop a line. 

Thank you in advance! Thomas

 

 

> Memory leak in DefaultCamelContext reported by Tomcat 10
> --------------------------------------------------------
>
>                 Key: CAMEL-17712
>                 URL: https://issues.apache.org/jira/browse/CAMEL-17712
>             Project: Camel
>          Issue Type: Bug
>          Components: camel-core-engine
>    Affects Versions: 3.14.1
>         Environment: Apache Camel 3.14.1
> Tomcat 10
> Java 17
>            Reporter: Thomas Hoffmann
>            Assignee: Claus Ibsen
>            Priority: Minor
>             Fix For: 3.11.6, 3.14.2, 3.16.0
>
>
> Hello,
> we are using Camel inside a Tomcat application. The application also supports reloading of the context.
> Unfortunately, we are getting a memory leak report, e.g.
> {code:java}
> [Catalina-utility-1] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [ROOT] created a ThreadLocal with key of type [java.lang.ThreadLocal.SuppliedThreadLocal] (value [java.lang.ThreadLocal$SuppliedThreadLocal@588bee00]) and a value of type [org.apache.camel.impl.DefaultCamelContext.OptionHolder] (value [org.apache.camel.impl.DefaultCamelContext$OptionHolder@1d5c4495]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.{code}
>  
> Looking at the code, the problem is within the java class DefaultCamelContext
> [https://github.com/apache/camel/blob/main/core/camel-core-engine/src/main/java/org/apache/camel/impl/DefaultCamelContext.java]
> It contains a class variable which is always instantiated with:
> ThreadLocal<OptionHolder> OPTIONS = ThreadLocal.withInitial(OptionHolder::new);
> This ThreadLocal is never cleaned up by OPTIONS.remove(), This should be added to the shutdown() or stop() handler maybe(?)
> Some additional null checks would have to be implemented as well to make it safe.
>  
> Thanks,
> Thomas



--
This message was sent by Atlassian Jira
(v8.20.1#820001)