You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tomcat.apache.org by bu...@apache.org on 2015/02/06 23:44:04 UTC
[Bug 57546] New: Memory Leak in SecureNioChannel
https://issues.apache.org/bugzilla/show_bug.cgi?id=57546
Bug ID: 57546
Summary: Memory Leak in SecureNioChannel
Product: Tomcat 8
Version: 8.0.18
Hardware: PC
OS: Windows NT
Status: NEW
Severity: major
Priority: P2
Component: WebSocket
Assignee: dev@tomcat.apache.org
Reporter: paul.gaylie@gmail.com
Created attachment 32440
--> https://issues.apache.org/bugzilla/attachment.cgi?id=32440&action=edit
Too many instances in hashmap
We are connecting mobile devices via websocket to Tomcat 8.0.18 server. The
physical mobile connection can die at random times. When the connection goes
down the instance of org.apache.coyote.http11.upgrade.NioProcessor is not
removed from the connections HashMap in class
org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler. There are
4,181 entries in the connections HashMap. However, there are in fact 1,400 real
socket connections. See DumpHprof1.jpg. This displays a drill-down into one of
the HashMap entries. The instance of SafeCommConnection is my ServerEndpoint
POJO. When onClose is called in my ServerEndpoint I set a flag called isClosing
to true. You can see on the right side that it is set to true so I have removed
it from my app but the instance remains in memory held by the NioProtocol's
connections HashMap. It seems that http11NioProtocol.release method is not
getting called. Also, here is a stack trace in the stderr log file. It gets
hundreds of these:
06-Feb-2015 14:12:05.255 INFO [http-nio-443-exec-385]
org.apache.tomcat.websocket.server.WsRemoteEndpointImplServer.doClose Failed to
close the ServletOutputStream connection cleanly
java.io.IOException: An existing connection was forcibly closed by the remote
host
at sun.nio.ch.SocketDispatcher.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(Unknown Source)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(Unknown Source)
at sun.nio.ch.IOUtil.write(Unknown Source)
at sun.nio.ch.SocketChannelImpl.write(Unknown Source)
at
org.apache.tomcat.util.net.SecureNioChannel.flush(SecureNioChannel.java:134)
at
org.apache.tomcat.util.net.SecureNioChannel.close(SecureNioChannel.java:370)
at
org.apache.tomcat.util.net.SecureNioChannel.close(SecureNioChannel.java:398)
at
org.apache.coyote.http11.upgrade.NioServletOutputStream.doClose(NioServletOutputStream.java:138)
at
org.apache.coyote.http11.upgrade.AbstractServletOutputStream.close(AbstractServletOutputStream.java:137)
at
org.apache.tomcat.websocket.server.WsRemoteEndpointImplServer.doClose(WsRemoteEndpointImplServer.java:143)
at
org.apache.tomcat.websocket.WsRemoteEndpointImplBase.close(WsRemoteEndpointImplBase.java:638)
at
org.apache.tomcat.websocket.server.WsRemoteEndpointImplServer.onWritePossible(WsRemoteEndpointImplServer.java:118)
at
org.apache.tomcat.websocket.server.WsRemoteEndpointImplServer.doWrite(WsRemoteEndpointImplServer.java:81)
at
org.apache.tomcat.websocket.WsRemoteEndpointImplBase.writeMessagePart(WsRemoteEndpointImplBase.java:450)
at
org.apache.tomcat.websocket.WsRemoteEndpointImplBase.startMessage(WsRemoteEndpointImplBase.java:338)
at
org.apache.tomcat.websocket.WsRemoteEndpointImplBase.startMessageBlock(WsRemoteEndpointImplBase.java:270)
at
org.apache.tomcat.websocket.WsSession.sendCloseMessage(WsSession.java:563)
at org.apache.tomcat.websocket.WsSession.onClose(WsSession.java:503)
at
org.apache.tomcat.websocket.server.WsHttpUpgradeHandler.close(WsHttpUpgradeHandler.java:183)
at
org.apache.tomcat.websocket.server.WsHttpUpgradeHandler.access$200(WsHttpUpgradeHandler.java:48)
at
org.apache.tomcat.websocket.server.WsHttpUpgradeHandler$WsReadListener.onDataAvailable(WsHttpUpgradeHandler.java:214)
at
org.apache.coyote.http11.upgrade.AbstractServletInputStream.onDataAvailable(AbstractServletInputStream.java:194)
at
org.apache.coyote.http11.upgrade.AbstractProcessor.upgradeDispatch(AbstractProcessor.java:96)
at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:654)
at
org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:223)
at
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1558)
at
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1515)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Unknown Source)
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[Bug 57546] Memory Leak in SecureNioChannel
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=57546
Konstantin Kolinko <kn...@gmail.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|RESOLVED |REOPENED
Component|WebSocket |WebSocket
Version|8.0.18 |7.0.59
Resolution|FIXED |---
Product|Tomcat 8 |Tomcat 7
Target Milestone|---- |---
--- Comment #6 from Konstantin Kolinko <kn...@gmail.com> ---
Only the first part of the fix was applied to Tomcat 7 (r1658738).
The second part was applied to Tomcat 8 (r1658794), but has not been ported to
Tomcat 7 yet. Thus I am REOPENing this issue, so that it is not forgotten.
The issues in the first part:
- as noted by Remy (in "Re: r1658734"):
> Good find, but what happens if onDataAvailable or onWritePossible throw a
runtime exception (like a NPE), since this is user code. It could also leak
then ? Shouldn't the code catch everything, also call onError on the
listener and close ?
- as noted by Chris (in "Re: r1658737"):
> Is it worth predicating the log call with if(getLog().isDebugEnabled)?
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[Bug 57546] Memory Leak in SecureNioChannel
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=57546
Mark Thomas <ma...@apache.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |RESOLVED
Resolution|--- |FIXED
--- Comment #5 from Mark Thomas <ma...@apache.org> ---
Thanks for the report and the pointers to reproduce this.
I have fixed the problem in trunk, 8.0.x (for 8.0.19 onwards) and 7.0.x (for
7.0.60 onwards).
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[Bug 57546] Memory Leak in SecureNioChannel
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=57546
--- Comment #3 from Mark Thomas <ma...@apache.org> ---
(In reply to Paul Gaylie from comment #2)
> Mark,
>
> Unfortunately, killing a browser is not a good simulation because the
> browser is smart enough to properly close the websocket when it gets killed.
With kill -9 the browser nver gets the chance. The close isn't clean - a bunch
of IOExceptions are triggered.
> Here is what to do: connect using a mobile browser on your phone (like
> Chrome on Android) and then shutoff the mobile data connection on your
> phone. Then you should see the doClose() operation happening on the server
> side.
I'll try but right now my money is on this already being fixed in the post
8.0.18 changes.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[Bug 57546] Memory Leak in SecureNioChannel
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=57546
--- Comment #2 from Paul <pa...@gmail.com> ---
Mark,
Unfortunately, killing a browser is not a good simulation because the browser
is smart enough to properly close the websocket when it gets killed.
Here is what to do: connect using a mobile browser on your phone (like Chrome
on Android) and then shutoff the mobile data connection on your phone. Then you
should see the doClose() operation happening on the server side.
Thanks for looking into this.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[Bug 57546] Memory Leak in SecureNioChannel
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=57546
--- Comment #1 from Mark Thomas <ma...@apache.org> ---
release() isn't expected to be called for an HTTP upgrade connection so that
method getting called is not part of the problem.
I've tried to reproduce this with 8.0.x trunk but I can't. I've been trying the
various WebSocket example applications with and without https and killing the
browser to simulate dropping the connection.
A simple test case that demonstrates this issue would be a big help.
I do wonder if some of the connector fixes made since 8.0.18 may have fixed
this. Are you able to build the latest 8.0.x code from svn and test that?
Alternatively I could provide you with a test build.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[Bug 57546] Memory Leak in SecureNioChannel
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=57546
--- Comment #4 from Mark Thomas <ma...@apache.org> ---
Interesting. I tried testing with a separate machine so I could pull out the
network cable. That has triggered a problem that looks like the one described
here. I'm looking into this at the moment.
Musing on why I didn't see this when I kill -9'd the browser, I suspect that
the client OS was doing some clean-up since it was still connected.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[Bug 57546] Memory Leak in SecureNioChannel
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=57546
--- Comment #9 from Mark Thomas <ma...@apache.org> ---
See r1658734 and r1658790.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[Bug 57546] Memory Leak in SecureNioChannel
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=57546
Mark Thomas <ma...@apache.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|REOPENED |RESOLVED
Resolution|--- |FIXED
--- Comment #7 from Mark Thomas <ma...@apache.org> ---
This wasn't forgotten it deliberately wasn't back-ported because this isn't
user code in Tomcat 7. It is a Tomcat internal API only used by WebSockets and
we know exactly which exceptions are thrown. Therefore the additional
protection added in trunk and 8.0.x is not necessary in 7.0.x.
Regarding the debug, the look-up in the StringManager is likely to be much
faster than the throwing of the Exception that triggers this look-up.
Performance is pretty much a lost cause once an Exception has been thrown.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[Bug 57546] Memory Leak in SecureNioChannel
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=57546
--- Comment #8 from Paul Gaylie <pa...@gmail.com> ---
Hi Mark,
Great that you found the bug.
Exactly what code needed to be updated? Just curious where the proper fix
needed to be applied - was driving me nuts.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org