You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tomcat.apache.org by bu...@apache.org on 2019/09/24 00:49:45 UTC
[Bug 63766] New: Resource leak: under certain conditions, request
objects related to WebSockets are not freed
https://bz.apache.org/bugzilla/show_bug.cgi?id=63766
Bug ID: 63766
Summary: Resource leak: under certain conditions, request
objects related to WebSockets are not freed
Product: Tomcat 8
Version: 8.5.38
Hardware: Macintosh
Status: NEW
Severity: major
Priority: P2
Component: WebSocket
Assignee: dev@tomcat.apache.org
Reporter: francis.vanaeken@servicenow.com
Target Milestone: ----
Created attachment 36794
--> https://bz.apache.org/bugzilla/attachment.cgi?id=36794&action=edit
Code to reproduce the problem (see description)
Resource leak: under certain conditions, request objects related to WebSockets
are not freed
When Tomcat 8.5.38 is setting up a WebSocket (WS) connection with a client (or
has just set up the connection - not sure), and then receives a TCP RST on that
connection, it is possible that the associated objects are never freed. The
objects are of the classes below.
org.apache.tomcat.websocket.server.WsHandshakeRequest
org.apache.catalina.connector.Request
org.apache.coyote.Request
org.apache.coyote.RequestInfo
org.apache.catalina.connector.RequestFacade
We saw this happen in production, and we were able to reproduce this with test
code, running against our application, and against an out-of-the-box (OOB)
(embedded) Tomcat.
I have attached the stack traces of the two use cases (our application and OOB
Tomcat). Interestingly, the stack traces are different.
To reproduce the problem in a test environment, we have modified a TCP proxy to
send a RST packet to the server shortly after sending the WebSocket upgrade
HTTP request. When opening many WS connections, and having them automatically
interrupted with RST packets, after a while a number of objects seem to be
stuck in memory (see screenshot requests_objects.png). The objects stay in
memory even when the proxy and client are shut down.
Thank you for having a look at this. This failure mode does not happen often,
but when it happens, it eventually can bring the JVM down because of memory
pressure.
ATTACHMENT
The attachment contains:
tomcat-webserver: an OOB (embedded) Tomcat with a WS endpoint
websockets/tcp-proxy: a TCP proxy, modified to send RST packets - run ProxyMain
to start the proxy
websockets/websockets-client: a simple WS client, opening many connections -
run SadPath to reproduce the problem
requests_objects.png: a VisualVM screenshot showing stuck objects
web_socket_connection_reset.txt: two stack traces (the first when reproducing
the problem with our application, the second when reproducing the problem with
tomcat-webserver)
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[Bug 63766] Resource leak: under certain conditions, request objects
related to WebSockets are not freed
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=63766
--- Comment #5 from Francis VAN AEKEN <fr...@servicenow.com> ---
Thank you for addressing this issue so swiftly. We are impressed. Keep up the
good work!
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[Bug 63766] Resource leak: under certain conditions, request objects
related to WebSockets are not freed
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=63766
Mark Thomas <ma...@apache.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
OS| |All
Status|NEW |NEEDINFO
--- Comment #1 from Mark Thomas <ma...@apache.org> ---
Please re-test this with the latest 8.5.x release (8.5.46 as I type this) and
confirm whether or not this resolves the issue.
If it does not resolve the issue please update the test case and we will
investigate.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[Bug 63766] Resource leak: under certain conditions, request objects
related to WebSockets are not freed
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=63766
--- Comment #2 from Francis VAN AEKEN <fr...@servicenow.com> ---
Created attachment 36798
--> https://bz.apache.org/bugzilla/attachment.cgi?id=36798&action=edit
Application code, using embedded Tomcat 8.5.46
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[Bug 63766] Resource leak: under certain conditions, request objects
related to WebSockets are not freed
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=63766
Mark Thomas <ma...@apache.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
Resolution|--- |FIXED
Status|NEW |RESOLVED
--- Comment #4 from Mark Thomas <ma...@apache.org> ---
Thanks for the report and the test case. It makes it so much easier to track
down and fix bugs when you have this much information to start from.
For the record, it was not what would normally a resource leak. More a failure
to clean-up in a timely manner. Given enough (error free) further processing,
the objects would have been cleaned up.
Fixed in:
- master for 9.0.27 onwards
- 8.5.x for 8.5.47 onwards
- 7.0.x for 7.0.97 onwards
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[Bug 63766] Resource leak: under certain conditions, request objects
related to WebSockets are not freed
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=63766
Francis VAN AEKEN <fr...@servicenow.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Version|8.5.38 |8.5.46
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[Bug 63766] Resource leak: under certain conditions, request objects
related to WebSockets are not freed
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=63766
Francis VAN AEKEN <fr...@servicenow.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEEDINFO |NEW
--- Comment #3 from Francis VAN AEKEN <fr...@servicenow.com> ---
We have retested using Tomcat 8.5.46 (new code attached), and could still
reproduce the problem, following the steps below.
Start the Tomcat application
Start the proxy
Start the test client (SadPath)
Wait for 20 minutes
Shut down the client
Shut down the proxy
Take a heapdump of the Tomcat application
In the heapdump, observe 100+ instances of each of the classes below
org.apache.coyote.Request
org.apache.catalina.connector.Request
org.apache.coyote.RequestInfo
org.apache.tomcat.websocket.server.WsHandshakeRequest
org.apache.catalina.connector.RequestFacade
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org