You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@wicket.apache.org by "Attila Király (JIRA)" <ji...@apache.org> on 2011/01/07 13:27:45 UTC
[jira] Created: (WICKET-3312) Tomcat complains about thread local
memory leaking when using wickets XForwardedRequestWrapperFactory
Tomcat complains about thread local memory leaking when using wickets XForwardedRequestWrapperFactory
-----------------------------------------------------------------------------------------------------
Key: WICKET-3312
URL: https://issues.apache.org/jira/browse/WICKET-3312
Project: Wicket
Issue Type: Bug
Affects Versions: 1.5-M3
Reporter: Attila Király
Priority: Minor
o.a.w.protocol.http.servlet.XForwardedRequestWrapperFactory can be used to wrap incoming servlet request into o.a.w.protocol.http.servlet.XForwardedRequestWrapper objects. These objects store a SimpleDateFormat array in a private static thread local memory but theye are never cleared. Tomcats memory leak prevention listener detects when the webapp is stopped. Tomcat 7.0.5 logs several times the followings:
2011.01.07. 12:50:59 org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
SEVERE: The web application [/wtl] created a ThreadLocal with key of type [null] (value [org.apache.wicket.protocol.http.servlet.XForwardedRequestWrapper$1@6328edf2]) and a value of type [java.text.SimpleDateFormat[]] (value [[Ljava.text.SimpleDateFormat;@141dddba]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (WICKET-3312) Tomcat complains about thread local
memory leaking when using wickets XForwardedRequestWrapperFactory
Posted by "Attila Király (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WICKET-3312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12982362#action_12982362 ]
Attila Király commented on WICKET-3312:
---------------------------------------
@Oleg Mikheev
It is not a tomcat issue. Tomcat only reports that you have a memory leak related error in your application. For more info you can read the links provided in a previous post or ask on the tomcat user list. This discussion is very off topic here, because this is the wicket frameworks issue tracker and your problem is not related to wicket.
> Tomcat complains about thread local memory leaking when using wickets XForwardedRequestWrapperFactory
> -----------------------------------------------------------------------------------------------------
>
> Key: WICKET-3312
> URL: https://issues.apache.org/jira/browse/WICKET-3312
> Project: Wicket
> Issue Type: Bug
> Affects Versions: 1.5-M3
> Reporter: Attila Király
> Assignee: Igor Vaynberg
> Priority: Minor
> Fix For: 1.5-RC1
>
> Attachments: quickstart.zip, WICKET-3312.patch
>
>
> o.a.w.protocol.http.servlet.XForwardedRequestWrapperFactory can be used to wrap incoming servlet request into o.a.w.protocol.http.servlet.XForwardedRequestWrapper objects. These objects store a SimpleDateFormat array in a private static thread local memory but theye are never cleared. Tomcats memory leak prevention listener detects when the webapp is stopped. Tomcat 7.0.5 logs several times the followings:
> 2011.01.07. 12:50:59 org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
> SEVERE: The web application [/wtl] created a ThreadLocal with key of type [null] (value [org.apache.wicket.protocol.http.servlet.XForwardedRequestWrapper$1@6328edf2]) and a value of type [java.text.SimpleDateFormat[]] (value [[Ljava.text.SimpleDateFormat;@141dddba]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (WICKET-3312) Tomcat complains about thread local
memory leaking when using wickets XForwardedRequestWrapperFactory
Posted by "Oleg Mikheev (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WICKET-3312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12982301#action_12982301 ]
Oleg Mikheev commented on WICKET-3312:
--------------------------------------
Guys, I'm seeing exactly the same issue on 6.0.26:
SEVERE: A web application created a ThreadLocal with key of type [null] (value [com.opensymphony.xwork2.inject.ContainerImpl$10@65ddc55f]) and a value of type [java.lang.Object[]] (value [[Ljava.lang.Object;@5d79a22d]) but failed to remove it when the web application was stopped. To prevent a memory leak, the ThreadLocal has been forcibly removed.
I'm keeping the EntityManager in ThreadLocal, and that message above floods catalina.out each time I shutdown Tomcat.
Does my issue seem to be the same as the one originally reported by Attila?
Do I understand it right that this issue is fixed in Tomcat 7 and not 6?
If yes then should I create a separate issue for 6?
Thanks
> Tomcat complains about thread local memory leaking when using wickets XForwardedRequestWrapperFactory
> -----------------------------------------------------------------------------------------------------
>
> Key: WICKET-3312
> URL: https://issues.apache.org/jira/browse/WICKET-3312
> Project: Wicket
> Issue Type: Bug
> Affects Versions: 1.5-M3
> Reporter: Attila Király
> Assignee: Igor Vaynberg
> Priority: Minor
> Fix For: 1.5-RC1
>
> Attachments: quickstart.zip, WICKET-3312.patch
>
>
> o.a.w.protocol.http.servlet.XForwardedRequestWrapperFactory can be used to wrap incoming servlet request into o.a.w.protocol.http.servlet.XForwardedRequestWrapper objects. These objects store a SimpleDateFormat array in a private static thread local memory but theye are never cleared. Tomcats memory leak prevention listener detects when the webapp is stopped. Tomcat 7.0.5 logs several times the followings:
> 2011.01.07. 12:50:59 org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
> SEVERE: The web application [/wtl] created a ThreadLocal with key of type [null] (value [org.apache.wicket.protocol.http.servlet.XForwardedRequestWrapper$1@6328edf2]) and a value of type [java.text.SimpleDateFormat[]] (value [[Ljava.text.SimpleDateFormat;@141dddba]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (WICKET-3312) Tomcat complains about thread local
memory leaking when using wickets XForwardedRequestWrapperFactory
Posted by "Igor Vaynberg (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WICKET-3312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Igor Vaynberg resolved WICKET-3312.
-----------------------------------
Resolution: Fixed
Fix Version/s: 1.5-M4
applied. i didnt realize the wrapper existed for a single request only. i dont think constructing those 3 formats will be too expensive.
i also didnt realize the leak was on undeploy, that makes sense. thats the piece you shouldve mentioned in your initial issue description :)
cheers.
> Tomcat complains about thread local memory leaking when using wickets XForwardedRequestWrapperFactory
> -----------------------------------------------------------------------------------------------------
>
> Key: WICKET-3312
> URL: https://issues.apache.org/jira/browse/WICKET-3312
> Project: Wicket
> Issue Type: Bug
> Affects Versions: 1.5-M3
> Reporter: Attila Király
> Assignee: Igor Vaynberg
> Priority: Minor
> Fix For: 1.5-M4
>
> Attachments: quickstart.zip, WICKET-3312.patch
>
>
> o.a.w.protocol.http.servlet.XForwardedRequestWrapperFactory can be used to wrap incoming servlet request into o.a.w.protocol.http.servlet.XForwardedRequestWrapper objects. These objects store a SimpleDateFormat array in a private static thread local memory but theye are never cleared. Tomcats memory leak prevention listener detects when the webapp is stopped. Tomcat 7.0.5 logs several times the followings:
> 2011.01.07. 12:50:59 org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
> SEVERE: The web application [/wtl] created a ThreadLocal with key of type [null] (value [org.apache.wicket.protocol.http.servlet.XForwardedRequestWrapper$1@6328edf2]) and a value of type [java.text.SimpleDateFormat[]] (value [[Ljava.text.SimpleDateFormat;@141dddba]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (WICKET-3312) Tomcat complains about thread local
memory leaking when using wickets XForwardedRequestWrapperFactory
Posted by "Igor Vaynberg (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WICKET-3312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Igor Vaynberg resolved WICKET-3312.
-----------------------------------
Resolution: Invalid
Assignee: Igor Vaynberg
can you provide some log output of this and a simple quickstart that reproduces it?
doesnt make sense that memory "leaks", it is a threadlocal so there will be as many entries as there are threads in the server's threadpool - but not more.
your patch is bad because simpledateformat is not threadsafe, so your patch will cause concurrency errors. if we remove the threadlocal we have to sync on the formats, but that is not as efficient as using a threadlocal.
i will close this until you provide the logs and/or a quickstart.
> Tomcat complains about thread local memory leaking when using wickets XForwardedRequestWrapperFactory
> -----------------------------------------------------------------------------------------------------
>
> Key: WICKET-3312
> URL: https://issues.apache.org/jira/browse/WICKET-3312
> Project: Wicket
> Issue Type: Bug
> Affects Versions: 1.5-M3
> Reporter: Attila Király
> Assignee: Igor Vaynberg
> Priority: Minor
> Attachments: WICKET-3312.patch
>
>
> o.a.w.protocol.http.servlet.XForwardedRequestWrapperFactory can be used to wrap incoming servlet request into o.a.w.protocol.http.servlet.XForwardedRequestWrapper objects. These objects store a SimpleDateFormat array in a private static thread local memory but theye are never cleared. Tomcats memory leak prevention listener detects when the webapp is stopped. Tomcat 7.0.5 logs several times the followings:
> 2011.01.07. 12:50:59 org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
> SEVERE: The web application [/wtl] created a ThreadLocal with key of type [null] (value [org.apache.wicket.protocol.http.servlet.XForwardedRequestWrapper$1@6328edf2]) and a value of type [java.text.SimpleDateFormat[]] (value [[Ljava.text.SimpleDateFormat;@141dddba]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Reopened: (WICKET-3312) Tomcat complains about thread local
memory leaking when using wickets XForwardedRequestWrapperFactory
Posted by "Igor Vaynberg (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WICKET-3312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Igor Vaynberg reopened WICKET-3312:
-----------------------------------
> Tomcat complains about thread local memory leaking when using wickets XForwardedRequestWrapperFactory
> -----------------------------------------------------------------------------------------------------
>
> Key: WICKET-3312
> URL: https://issues.apache.org/jira/browse/WICKET-3312
> Project: Wicket
> Issue Type: Bug
> Affects Versions: 1.5-M3
> Reporter: Attila Király
> Assignee: Igor Vaynberg
> Priority: Minor
> Attachments: quickstart.zip, WICKET-3312.patch
>
>
> o.a.w.protocol.http.servlet.XForwardedRequestWrapperFactory can be used to wrap incoming servlet request into o.a.w.protocol.http.servlet.XForwardedRequestWrapper objects. These objects store a SimpleDateFormat array in a private static thread local memory but theye are never cleared. Tomcats memory leak prevention listener detects when the webapp is stopped. Tomcat 7.0.5 logs several times the followings:
> 2011.01.07. 12:50:59 org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
> SEVERE: The web application [/wtl] created a ThreadLocal with key of type [null] (value [org.apache.wicket.protocol.http.servlet.XForwardedRequestWrapper$1@6328edf2]) and a value of type [java.text.SimpleDateFormat[]] (value [[Ljava.text.SimpleDateFormat;@141dddba]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (WICKET-3312) Tomcat complains about thread local
memory leaking when using wickets XForwardedRequestWrapperFactory
Posted by "Attila Király (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WICKET-3312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Attila Király updated WICKET-3312:
----------------------------------
Attachment: quickstart.zip
Please reopen this issue.
I. Quickstart
I am attaching a quickstart maven project. To reproduce do the followings:
1. Unzip quickstart.zip, run "mvn package" from wt directory to generate wt.war in target directory.
2. Start up a vanilla Tomcat installation (6.0.29 and 7.0.5 are both fine).
3. Copy wt.war into the webapps directory of tomcat. After a few seconds Tomcat will notice the new war and deploy it (unzip into wt directory). This can be seen on console / in log.
4. Visit http://localhost:8080/wt/index.html . You should see "Hello World"
5. Delete wt.war from tomcats webapps directory. After a few seconds Tomcat will notice this change and undeploy the war (delete wt directory). When this is done Tomcat will log the following:
SEVERE: The web application [/wt] created a ThreadLocal with key of type [null] (value [org.apache.wicket.protocol.http.servlet.XForwardedRequestWrapper$1@7dc21ece]) and a value of type [java.text.SimpleDateFormat[]] (value [[Ljava.text.SimpleDateFormat;@5374a6e2]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.
II. The reason for this leak is the following (imho).
Tomcat (and probably other servlet containers too) work like these:
- They have a common thread pool. The threads in this pool are used to serve request for all web applications. The lifecycle of the threads is different than the webapps.
- Each webapp has its own Web Application Classloader instance. This WAC is used to load the classes bundled with the webapp.
XForwardedRequestWrapper defines and uses a ThreadLocal subclass (let us call it S). S is bundled inside the wicket jars with the webapp so it is loaded by the WAC. The instances of S are hold in the thread locals of threads. When the application is undeployed/redeployed Tomcat drops the references to WAC in the hope the GC will reclaim the memory of all loaded classes by this loader. However the threads live and in the thread local an instance of S is referenced. This instance holds a reference to the S class. The S class holds reference to the WAC object. The WAC object holds references to every classes it loaded. This is the reason for the leak. And because we are talking about classes this is a leak in the permgen space not in the heap. Tomcat detects and clears it but not all servlet containers do this. For more info see Tomcat wiki [1].
Logback had a similar issue last year they fixed it too. See [2] & [3] for it.
III. About the thread safety of my patch
Yes SimpleDateFormat is not thread safe, I know that. My patch moves the SimpleDateFormat array from a static variable to an instance variable of a servlet request wrapper object. My assumption was when I made the test that servlet requests and servlet request wrappers do not have to be thread safe because they can only be used by one thread at a time.
Servlet spec 3.0 states the same:
"2.3.3.4
Thread Safety
Other than the startAsync and complete methods, implementations of the
request and response objects are not guaranteed to be thread safe. This means that
they should either only be used within the scope of the request handling thread or
the application must ensure that access to the request and response objects are
thread safe.
If a thread created by the application uses the container-managed objects, such as
the request or response object, those objects must be accessed only within the
object's life cycle as defined in sections 3.10 and 5.6. Be aware that other than the
startAsync, and complete methods, the request and response objects are not
thread safe. If those objects were accessed in the multiple threads, the access should
be synchronized or be done through a wrapper to add the thread safety, for instance,
synchronizing the call of the methods to access the request attribute, or using a local
output stream for the response object within a thread."
Also some other libraries think the same about request wrappers.
Jiras CapturingRequestWrapper [4]:
"THREAD SAFETY - One of these must be constructed on each HTTP request/response and hence only one thread may write to it at the one time. This suits the whole servlet engine idea,"
Spring security 2.0 SavedRequestAwareWrapper uses a SimpleDateFormat array [5] (this class is not present in the 3.0.x branch):
"protected SimpleDateFormat[] formats
The set of SimpleDateFormat formats to use in getDateHeader(). Notice that because SimpleDateFormat is not thread-safe, we can't declare formats[] as a static variable."
[1] Tomcat wiki about memory leaks: http://wiki.apache.org/tomcat/MemoryLeakProtection#customThreadLocal
[2] Logback issue about UnsynchronizedAppenderBase: http://jira.qos.ch/browse/LBCLASSIC-183
[3] Logback git diff for UnsynchronizedAppenderBase (ThreadLocal subclass holds Boolean): https://github.com/ceki/logback/commit/93995946b59b3ed1622d92c06b39c21942d98c58
[4] Jiras CapturingRequestWrapper: http://docs.atlassian.com/software/jira/docs/api/latest/com/atlassian/jira/util/http/request/CapturingRequestWrapper.html
[5] SavedRequestAwareWrapper class in Spring security 2.0: http://static.springsource.org/spring-security/site/docs/2.0.x/apidocs/org/springframework/security/wrapper/SavedRequestAwareWrapper.html#formats
> Tomcat complains about thread local memory leaking when using wickets XForwardedRequestWrapperFactory
> -----------------------------------------------------------------------------------------------------
>
> Key: WICKET-3312
> URL: https://issues.apache.org/jira/browse/WICKET-3312
> Project: Wicket
> Issue Type: Bug
> Affects Versions: 1.5-M3
> Reporter: Attila Király
> Assignee: Igor Vaynberg
> Priority: Minor
> Attachments: quickstart.zip, WICKET-3312.patch
>
>
> o.a.w.protocol.http.servlet.XForwardedRequestWrapperFactory can be used to wrap incoming servlet request into o.a.w.protocol.http.servlet.XForwardedRequestWrapper objects. These objects store a SimpleDateFormat array in a private static thread local memory but theye are never cleared. Tomcats memory leak prevention listener detects when the webapp is stopped. Tomcat 7.0.5 logs several times the followings:
> 2011.01.07. 12:50:59 org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
> SEVERE: The web application [/wtl] created a ThreadLocal with key of type [null] (value [org.apache.wicket.protocol.http.servlet.XForwardedRequestWrapper$1@6328edf2]) and a value of type [java.text.SimpleDateFormat[]] (value [[Ljava.text.SimpleDateFormat;@141dddba]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (WICKET-3312) Tomcat complains about thread local
memory leaking when using wickets XForwardedRequestWrapperFactory
Posted by "Attila Király (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WICKET-3312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Attila Király updated WICKET-3312:
----------------------------------
Attachment: WICKET-3312.patch
Attaching a patch against current trunk. Tests pass, tomcat does not complain anymore.
Patch simply drops static threadlocal variable and uses instance variable to hold the SimpleDateFormat array.
> Tomcat complains about thread local memory leaking when using wickets XForwardedRequestWrapperFactory
> -----------------------------------------------------------------------------------------------------
>
> Key: WICKET-3312
> URL: https://issues.apache.org/jira/browse/WICKET-3312
> Project: Wicket
> Issue Type: Bug
> Affects Versions: 1.5-M3
> Reporter: Attila Király
> Priority: Minor
> Attachments: WICKET-3312.patch
>
>
> o.a.w.protocol.http.servlet.XForwardedRequestWrapperFactory can be used to wrap incoming servlet request into o.a.w.protocol.http.servlet.XForwardedRequestWrapper objects. These objects store a SimpleDateFormat array in a private static thread local memory but theye are never cleared. Tomcats memory leak prevention listener detects when the webapp is stopped. Tomcat 7.0.5 logs several times the followings:
> 2011.01.07. 12:50:59 org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
> SEVERE: The web application [/wtl] created a ThreadLocal with key of type [null] (value [org.apache.wicket.protocol.http.servlet.XForwardedRequestWrapper$1@6328edf2]) and a value of type [java.text.SimpleDateFormat[]] (value [[Ljava.text.SimpleDateFormat;@141dddba]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.