You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Kirill Ilyukhin <ki...@gmail.com> on 2019/07/26 09:33:47 UTC
Resource leak for WebSockets closed with NOT_CONSISTENT reason
Hello,
When Tomcat receives WebSocket text message with invalid UTF-8, it closes
this connection with NOT_CONSISTENT reason. But after that some objects
(WsSession, UpgradeHandler, etc) stay in heap forever. They are referenced
from AbstractProtocol's connections map.
This leak consistently happens with Tomcat 8.5.3 and 8.5.43, both on Mac OS
and Windows, with or without Tomcat native.
I have created a very simple WebSocket Endpoint which does nothing except
logging its events and incoming messages, please see the code below. Also
you will need a WebSocket client which sends broken UTF-8 in text message
right after connecting to the server.
Thank you,
Kirill
-----------------------
package com.example.wstest;
import org.apache.log4j.Logger;
import javax.websocket.*;
public class WSEndpoint extends Endpoint {
private static final Logger logger = Logger.getLogger(WSEndpoint.class);
private WSConnection connection;
@Override
public void onOpen(Session session, EndpointConfig config) {
connection = new WSConnection(session);
logger.info("Opened WebSocket session-" + session.getId());
}
@Override
public void onClose(Session session, CloseReason closeReason) {
logger.info("Closed WebSocket session-" + session.getId() + ",
reason: " + closeReason.getCloseCode() + " (" +
closeReason.getReasonPhrase() + ")");
connection.destroy();
connection = null;
}
@Override
public void onError(Session session, Throwable throwable) {
logger.info("Error on WebSocket session-" + session.getId(),
throwable);
connection.destroy();
connection = null;
}
static class WSConnection implements MessageHandler.Whole<String> {
private final Session session;
WSConnection(Session session) {
this.session = session;
session.addMessageHandler(this);
}
public void destroy() {
session.removeMessageHandler(this);
}
@Override
public void onMessage(String message) {
logger.info("Session-" + session.getId() + " onMessage(" +
message +")");
}
}
}
-----------------------
Re: Resource leak for WebSockets closed with NOT_CONSISTENT reason
Posted by Mark Thomas <ma...@apache.org>.
On 31/07/2019 04:37, Kirill Ilyukhin wrote:
> Hello Mark,
>
> Please let me disagree with you. I have connected JVisualVM to Tomcat JVM,
> run the test with 10,000 connections, performed several explicit GCs and
> made a heap dump. All 10,000 WsSessions are in heap. They are referenced as
> following:
Sorry, I can't repeat this. I am using:
- the sample server code provided previously
- the client code provided previously
- the server.xml provided below
- Tomcat 8.5.44-dev (there are no relevant changes since 8.5.43)
- openjdk 1.8.0_212-b04
- linux
I have also tried to re-create the issue on OSX without success.
I have also tried to re-create the issue with 8.5.43 without success.
> this
> <- wsSession - class:
> org.apache.tomcat.websocket.server.WsHttpUpgradeHandler
> <- internalHttpUpgradeHandler - class:
> org.apache.coyote.http11.upgrade.UpgradeProcessorInternal
> <- val - class: java.util.concurrent.ConcurrentHashMap$Node
> <- [12930] - class: java.util.concurrent.ConcurrentHashMap$Node[]
> <- table - class: java.util.concurrent.ConcurrentHashMap
> <- connections - class:
> org.apache.coyote.AbstractProtocol$ConnectionHandler
> <- handler - class: org.apache.tomcat.util.net.NioEndpoint
> <- this$0 (Java frame) - class:
> org.apache.tomcat.util.net.NioEndpoint$Acceptor
That suggests connections aren't being closed properly. Various bugs
have been fixed in this area but all those fixes are present in 8.5.43.
> Do you think it might be Tomcat configuration dependent? Please see my
> server.xml below.
I think there is something about your environment that is triggering
this. Have you tried re-creating this with a completely clean (OS, Java,
Tomcat) install?
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Resource leak for WebSockets closed with NOT_CONSISTENT reason
Posted by Kirill Ilyukhin <ki...@gmail.com>.
Hello Mark,
Please let me disagree with you. I have connected JVisualVM to Tomcat JVM,
run the test with 10,000 connections, performed several explicit GCs and
made a heap dump. All 10,000 WsSessions are in heap. They are referenced as
following:
this
<- wsSession - class:
org.apache.tomcat.websocket.server.WsHttpUpgradeHandler
<- internalHttpUpgradeHandler - class:
org.apache.coyote.http11.upgrade.UpgradeProcessorInternal
<- val - class: java.util.concurrent.ConcurrentHashMap$Node
<- [12930] - class: java.util.concurrent.ConcurrentHashMap$Node[]
<- table - class: java.util.concurrent.ConcurrentHashMap
<- connections - class:
org.apache.coyote.AbstractProtocol$ConnectionHandler
<- handler - class: org.apache.tomcat.util.net.NioEndpoint
<- this$0 (Java frame) - class:
org.apache.tomcat.util.net.NioEndpoint$Acceptor
Do you think it might be Tomcat configuration dependent? Please see my
server.xml below.
<?xml version='1.0' encoding='utf-8'?>
<Server port="8005" shutdown="SHUTDOWN">
<Listener className="org.apache.catalina.startup.VersionLoggerListener" />
<Listener className="org.apache.catalina.core.AprLifecycleListener"
SSLEngine="on" />
<Listener
className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />
<Listener
className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />
<Listener
className="org.apache.catalina.core.ThreadLocalLeakPreventionListener" />
<GlobalNamingResources>
<Resource name="UserDatabase" auth="Container"
type="org.apache.catalina.UserDatabase"
description="User database that can be updated and saved"
factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
pathname="conf/tomcat-users.xml" />
</GlobalNamingResources>
<Service name="Catalina">
<Connector port="8080"
protocol="org.apache.coyote.http11.Http11NioProtocol"
asyncTimeout="20000"
URIEncoding="utf-8"
acceptorThreadCount="2"
compressableMimeType="text/html,text/xml,text/plain,text/x-json,application/javascript,application/json,text/css"
compression="on"
connectionTimeout="60000"
maxThreads="1024"
processorCache="512"
redirectPort="8443" />
<Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />
<Engine name="Catalina" defaultHost="localhost">
<Realm className="org.apache.catalina.realm.LockOutRealm">
<Realm className="org.apache.catalina.realm.UserDatabaseRealm"
resourceName="UserDatabase"/>
</Realm>
<Host name="localhost" appBase="webapps"
unpackWARs="true" autoDeploy="true">
<Valve className="org.apache.catalina.valves.AccessLogValve"
directory="logs"
prefix="localhost_access_log." suffix=".txt"
pattern="%h %l %u %t "%r"
"%{User-Agent}i" %s %b %Dms" resolveHosts="false"/>
</Host>
</Engine>
</Service>
</Server>
Thank you,
Kirill
On Tue, 30 Jul 2019 at 21:45, Mark Thomas <ma...@apache.org> wrote:
> On 30/07/2019 05:48, Kirill Ilyukhin wrote:
> > Hello Mark,
> >
> > Please see the test case and Tomcat JVM heap dump screenshot attached.
> > For sake of simplicity I do Thread.sleep() in client code instead of
> > reading bytes from server.
>
> That test case does not demonstrate a memory leak in either Tomcat 9.0.x
> nor Tomcat 8.5.x.
>
> There are some WsSession instances in memory at the end of the test but
> they are unreachable - i.e. eligible for GC.
>
> Mark
>
>
> > Test configuration is the following:
> >
> > Server version: Apache Tomcat/8.5.3
> > Server built: Jun 9 2016 11:16:29 UTC
> > Server number: 8.5.3.0
> > OS Name: Mac OS X
> > OS Version: 10.14.5
> > Architecture: x86_64
> > JVM Version: 1.8.0_112-b16
> > JVM Vendor: Oracle Corporation
> > The APR based Apache Tomcat Native library which allows optimal
> > performance in production environments was not found on the
> > java.library.path: ...
> >
> > Thank you,
> > Kirill
> >
> > On Tue, 30 Jul 2019 at 02:15, Mark Thomas <markt@apache.org
> > <ma...@apache.org>> wrote:
> >
> > On 26/07/2019 10:33, Kirill Ilyukhin wrote:
> > > Hello,
> > >
> > > When Tomcat receives WebSocket text message with invalid UTF-8, it
> > closes
> > > this connection with NOT_CONSISTENT reason. But after that some
> > objects
> > > (WsSession, UpgradeHandler, etc) stay in heap forever. They are
> > referenced
> > > from AbstractProtocol's connections map.
> > >
> > > This leak consistently happens with Tomcat 8.5.3 and 8.5.43, both
> > on Mac OS
> > > and Windows, with or without Tomcat native.
> > >
> > > I have created a very simple WebSocket Endpoint which does nothing
> > except
> > > logging its events and incoming messages, please see the code
> > below. Also
> > > you will need a WebSocket client which sends broken UTF-8 in text
> > message
> > > right after connecting to the server.
> >
> > I can't repeat this with either 9.0.x nor 8.5.x. I've repeated the
> steps
> > described above and checked the resulting state with a profiler. No
> > references are retained to WsSession objects nor WsHttpUpgradeHandler
> > objects.
> >
> > You'll need to provide the simplest possible test case (single class
> > client, simplest possible WAR) that demonstrates the issue.
> >
> > Mark
> >
> >
> > >
> > > Thank you,
> > > Kirill
> > >
> > > -----------------------
> > > package com.example.wstest;
> > >
> > > import org.apache.log4j.Logger;
> > > import javax.websocket.*;
> > >
> > > public class WSEndpoint extends Endpoint {
> > > private static final Logger logger =
> > Logger.getLogger(WSEndpoint.class);
> > > private WSConnection connection;
> > >
> > > @Override
> > > public void onOpen(Session session, EndpointConfig config) {
> > > connection = new WSConnection(session);
> > > logger.info <http://logger.info>("Opened WebSocket
> > session-" + session.getId());
> > > }
> > >
> > > @Override
> > > public void onClose(Session session, CloseReason closeReason) {
> > > logger.info <http://logger.info>("Closed WebSocket
> > session-" + session.getId() + ",
> > > reason: " + closeReason.getCloseCode() + " (" +
> > > closeReason.getReasonPhrase() + ")");
> > > connection.destroy();
> > > connection = null;
> > > }
> > >
> > > @Override
> > > public void onError(Session session, Throwable throwable) {
> > > logger.info <http://logger.info>("Error on WebSocket
> > session-" + session.getId(),
> > > throwable);
> > > connection.destroy();
> > > connection = null;
> > > }
> > >
> > > static class WSConnection implements
> > MessageHandler.Whole<String> {
> > > private final Session session;
> > >
> > > WSConnection(Session session) {
> > > this.session = session;
> > > session.addMessageHandler(this);
> > > }
> > >
> > > public void destroy() {
> > > session.removeMessageHandler(this);
> > > }
> > >
> > > @Override
> > > public void onMessage(String message) {
> > > logger.info <http://logger.info>("Session-" +
> > session.getId() + " onMessage(" +
> > > message +")");
> > > }
> > > }
> > > }
> > > -----------------------
> > >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> > <ma...@tomcat.apache.org>
> > For additional commands, e-mail: users-help@tomcat.apache.org
> > <ma...@tomcat.apache.org>
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> > For additional commands, e-mail: users-help@tomcat.apache.org
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: Resource leak for WebSockets closed with NOT_CONSISTENT reason
Posted by Mark Thomas <ma...@apache.org>.
On 30/07/2019 05:48, Kirill Ilyukhin wrote:
> Hello Mark,
>
> Please see the test case and Tomcat JVM heap dump screenshot attached.
> For sake of simplicity I do Thread.sleep() in client code instead of
> reading bytes from server.
That test case does not demonstrate a memory leak in either Tomcat 9.0.x
nor Tomcat 8.5.x.
There are some WsSession instances in memory at the end of the test but
they are unreachable - i.e. eligible for GC.
Mark
> Test configuration is the following:
>
> Server version: Apache Tomcat/8.5.3
> Server built: Jun 9 2016 11:16:29 UTC
> Server number: 8.5.3.0
> OS Name: Mac OS X
> OS Version: 10.14.5
> Architecture: x86_64
> JVM Version: 1.8.0_112-b16
> JVM Vendor: Oracle Corporation
> The APR based Apache Tomcat Native library which allows optimal
> performance in production environments was not found on the
> java.library.path: ...
>
> Thank you,
> Kirill
>
> On Tue, 30 Jul 2019 at 02:15, Mark Thomas <markt@apache.org
> <ma...@apache.org>> wrote:
>
> On 26/07/2019 10:33, Kirill Ilyukhin wrote:
> > Hello,
> >
> > When Tomcat receives WebSocket text message with invalid UTF-8, it
> closes
> > this connection with NOT_CONSISTENT reason. But after that some
> objects
> > (WsSession, UpgradeHandler, etc) stay in heap forever. They are
> referenced
> > from AbstractProtocol's connections map.
> >
> > This leak consistently happens with Tomcat 8.5.3 and 8.5.43, both
> on Mac OS
> > and Windows, with or without Tomcat native.
> >
> > I have created a very simple WebSocket Endpoint which does nothing
> except
> > logging its events and incoming messages, please see the code
> below. Also
> > you will need a WebSocket client which sends broken UTF-8 in text
> message
> > right after connecting to the server.
>
> I can't repeat this with either 9.0.x nor 8.5.x. I've repeated the steps
> described above and checked the resulting state with a profiler. No
> references are retained to WsSession objects nor WsHttpUpgradeHandler
> objects.
>
> You'll need to provide the simplest possible test case (single class
> client, simplest possible WAR) that demonstrates the issue.
>
> Mark
>
>
> >
> > Thank you,
> > Kirill
> >
> > -----------------------
> > package com.example.wstest;
> >
> > import org.apache.log4j.Logger;
> > import javax.websocket.*;
> >
> > public class WSEndpoint extends Endpoint {
> > private static final Logger logger =
> Logger.getLogger(WSEndpoint.class);
> > private WSConnection connection;
> >
> > @Override
> > public void onOpen(Session session, EndpointConfig config) {
> > connection = new WSConnection(session);
> > logger.info <http://logger.info>("Opened WebSocket
> session-" + session.getId());
> > }
> >
> > @Override
> > public void onClose(Session session, CloseReason closeReason) {
> > logger.info <http://logger.info>("Closed WebSocket
> session-" + session.getId() + ",
> > reason: " + closeReason.getCloseCode() + " (" +
> > closeReason.getReasonPhrase() + ")");
> > connection.destroy();
> > connection = null;
> > }
> >
> > @Override
> > public void onError(Session session, Throwable throwable) {
> > logger.info <http://logger.info>("Error on WebSocket
> session-" + session.getId(),
> > throwable);
> > connection.destroy();
> > connection = null;
> > }
> >
> > static class WSConnection implements
> MessageHandler.Whole<String> {
> > private final Session session;
> >
> > WSConnection(Session session) {
> > this.session = session;
> > session.addMessageHandler(this);
> > }
> >
> > public void destroy() {
> > session.removeMessageHandler(this);
> > }
> >
> > @Override
> > public void onMessage(String message) {
> > logger.info <http://logger.info>("Session-" +
> session.getId() + " onMessage(" +
> > message +")");
> > }
> > }
> > }
> > -----------------------
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> <ma...@tomcat.apache.org>
> For additional commands, e-mail: users-help@tomcat.apache.org
> <ma...@tomcat.apache.org>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Resource leak for WebSockets closed with NOT_CONSISTENT reason
Posted by Kirill Ilyukhin <ki...@gmail.com>.
Hello Mark,
Please see the test case and Tomcat JVM heap dump screenshot attached.
For sake of simplicity I do Thread.sleep() in client code instead of
reading bytes from server.
Test configuration is the following:
Server version: Apache Tomcat/8.5.3
Server built: Jun 9 2016 11:16:29 UTC
Server number: 8.5.3.0
OS Name: Mac OS X
OS Version: 10.14.5
Architecture: x86_64
JVM Version: 1.8.0_112-b16
JVM Vendor: Oracle Corporation
The APR based Apache Tomcat Native library which allows optimal performance
in production environments was not found on the java.library.path: ...
Thank you,
Kirill
On Tue, 30 Jul 2019 at 02:15, Mark Thomas <ma...@apache.org> wrote:
> On 26/07/2019 10:33, Kirill Ilyukhin wrote:
> > Hello,
> >
> > When Tomcat receives WebSocket text message with invalid UTF-8, it closes
> > this connection with NOT_CONSISTENT reason. But after that some objects
> > (WsSession, UpgradeHandler, etc) stay in heap forever. They are
> referenced
> > from AbstractProtocol's connections map.
> >
> > This leak consistently happens with Tomcat 8.5.3 and 8.5.43, both on Mac
> OS
> > and Windows, with or without Tomcat native.
> >
> > I have created a very simple WebSocket Endpoint which does nothing except
> > logging its events and incoming messages, please see the code below. Also
> > you will need a WebSocket client which sends broken UTF-8 in text message
> > right after connecting to the server.
>
> I can't repeat this with either 9.0.x nor 8.5.x. I've repeated the steps
> described above and checked the resulting state with a profiler. No
> references are retained to WsSession objects nor WsHttpUpgradeHandler
> objects.
>
> You'll need to provide the simplest possible test case (single class
> client, simplest possible WAR) that demonstrates the issue.
>
> Mark
>
>
> >
> > Thank you,
> > Kirill
> >
> > -----------------------
> > package com.example.wstest;
> >
> > import org.apache.log4j.Logger;
> > import javax.websocket.*;
> >
> > public class WSEndpoint extends Endpoint {
> > private static final Logger logger =
> Logger.getLogger(WSEndpoint.class);
> > private WSConnection connection;
> >
> > @Override
> > public void onOpen(Session session, EndpointConfig config) {
> > connection = new WSConnection(session);
> > logger.info("Opened WebSocket session-" + session.getId());
> > }
> >
> > @Override
> > public void onClose(Session session, CloseReason closeReason) {
> > logger.info("Closed WebSocket session-" + session.getId() + ",
> > reason: " + closeReason.getCloseCode() + " (" +
> > closeReason.getReasonPhrase() + ")");
> > connection.destroy();
> > connection = null;
> > }
> >
> > @Override
> > public void onError(Session session, Throwable throwable) {
> > logger.info("Error on WebSocket session-" + session.getId(),
> > throwable);
> > connection.destroy();
> > connection = null;
> > }
> >
> > static class WSConnection implements MessageHandler.Whole<String> {
> > private final Session session;
> >
> > WSConnection(Session session) {
> > this.session = session;
> > session.addMessageHandler(this);
> > }
> >
> > public void destroy() {
> > session.removeMessageHandler(this);
> > }
> >
> > @Override
> > public void onMessage(String message) {
> > logger.info("Session-" + session.getId() + " onMessage(" +
> > message +")");
> > }
> > }
> > }
> > -----------------------
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: Resource leak for WebSockets closed with NOT_CONSISTENT reason
Posted by Mark Thomas <ma...@apache.org>.
On 26/07/2019 10:33, Kirill Ilyukhin wrote:
> Hello,
>
> When Tomcat receives WebSocket text message with invalid UTF-8, it closes
> this connection with NOT_CONSISTENT reason. But after that some objects
> (WsSession, UpgradeHandler, etc) stay in heap forever. They are referenced
> from AbstractProtocol's connections map.
>
> This leak consistently happens with Tomcat 8.5.3 and 8.5.43, both on Mac OS
> and Windows, with or without Tomcat native.
>
> I have created a very simple WebSocket Endpoint which does nothing except
> logging its events and incoming messages, please see the code below. Also
> you will need a WebSocket client which sends broken UTF-8 in text message
> right after connecting to the server.
I can't repeat this with either 9.0.x nor 8.5.x. I've repeated the steps
described above and checked the resulting state with a profiler. No
references are retained to WsSession objects nor WsHttpUpgradeHandler
objects.
You'll need to provide the simplest possible test case (single class
client, simplest possible WAR) that demonstrates the issue.
Mark
>
> Thank you,
> Kirill
>
> -----------------------
> package com.example.wstest;
>
> import org.apache.log4j.Logger;
> import javax.websocket.*;
>
> public class WSEndpoint extends Endpoint {
> private static final Logger logger = Logger.getLogger(WSEndpoint.class);
> private WSConnection connection;
>
> @Override
> public void onOpen(Session session, EndpointConfig config) {
> connection = new WSConnection(session);
> logger.info("Opened WebSocket session-" + session.getId());
> }
>
> @Override
> public void onClose(Session session, CloseReason closeReason) {
> logger.info("Closed WebSocket session-" + session.getId() + ",
> reason: " + closeReason.getCloseCode() + " (" +
> closeReason.getReasonPhrase() + ")");
> connection.destroy();
> connection = null;
> }
>
> @Override
> public void onError(Session session, Throwable throwable) {
> logger.info("Error on WebSocket session-" + session.getId(),
> throwable);
> connection.destroy();
> connection = null;
> }
>
> static class WSConnection implements MessageHandler.Whole<String> {
> private final Session session;
>
> WSConnection(Session session) {
> this.session = session;
> session.addMessageHandler(this);
> }
>
> public void destroy() {
> session.removeMessageHandler(this);
> }
>
> @Override
> public void onMessage(String message) {
> logger.info("Session-" + session.getId() + " onMessage(" +
> message +")");
> }
> }
> }
> -----------------------
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org