You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Chen Levy <ch...@hotmail.com> on 2019/08/05 20:49:00 UTC
HTTP2 Connection Timeout
Hello Experts
Several of my production servers were recently upgraded from Tomcat 9.0.14 to 9.0.21; immediately after the upgrade the servers started accumulating memory and open-files (on Linux) in a steady trend that was not observed before.
After a couple of days (without reaching the memory or open-files limit and without throwing "OutOfMemoryError: Java heap space" or "IOException: Too many open files") the servers became unresponsive: any HTTPS request timed-out while HTTP requests continued to work correctly.
Restarting the servers resolved the symptoms but the behavior persists and a restart is necessary every couple of days.
I loaded a heap dump from an unresponsive server into MAT and received the following Leak Suspect:
105,871 instances of "org.apache.coyote.http2.Stream", loaded by "java.net.URLClassLoader..." occupy 7,581,549,904 (80.68%) bytes.
These instances are referenced from one instance of "java.util.concurrent.ConcurrentHashMap$Node[]", loaded by "<system class loader>"
The HashMap referenced in the report is "connections" inside ConnectionHandler.
I suspect that these objects accumulate as clients may not close their connections correctly; regardless, I'd expect Tomcat to close the connections upon timeout.
With keepAliveTimeout="20000" defined on UpgradeProtocol, I tested one simple HTTP2 connection's persistence on Chrome's net-internals.
With 9.0.14 I can see the following at 20 seconds (as expected):
...
t=7065701 [st= 64] HTTP2_SESSION_UPDATE_RECV_WINDOW
--> delta = 6894
--> window_size = 15728640
t=7085708 [st=20071] HTTP2_SESSION_PING
--> is_ack = false
--> type = "received"
--> unique_id = 2
t=7085708 [st=20071] HTTP2_SESSION_PING
--> is_ack = true
--> type = "sent"
--> unique_id = 2
t=7085708 [st=20071] HTTP2_SESSION_CLOSE
--> description = "Connection closed"
--> net_error = -100 (ERR_CONNECTION_CLOSED)
t=7085708 [st=20071] HTTP2_SESSION_POOL_REMOVE_SESSION
t=7085708 [st=20071] -HTTP2_SESSION
With 9.0.21 the connection does not close, even after several minutes.
I believe the change in behavior stems the following commit: https://github.com/apache/tomcat/commit/c16d9d810a1f64cd768ff33058936cf8907e3117 and so I may be doing something wrong.
Please let me know whether I have misconfigured, misunderstood, misdiagnosed, misbehaved or mis-something-else, and whether I should provide additional information
Current setup of the production servers:
AdoptOpenJDK (build 11.0.3+7)
Amazon Linux 2
<Connector port="443" protocol="org.apache.coyote.http11.Http11NioProtocol"
maxHttpHeaderSize="16384"
maxThreads="500" minSpareThreads="25"
enableLookups="false" disableUploadTimeout="true"
connectionTimeout="10000"
compression="on"
SSLEnabled="true" scheme="https" secure="true">
<UpgradeProtocol className="org.apache.coyote.http2.Http2Protocol"
keepAliveTimeout="20000"/>
<SSLHostConfig protocols="+TLSv1.2+TLSv1.3">
<Certificate certificateKeystoreFile="tomcat.keystore"
certificateKeyAlias="tomcat"
certificateKeystorePassword=""
certificateKeystoreType="PKCS12"/>
</SSLHostConfig>
</Connector>
Thanks
Chen
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: HTTP2 Connection Timeout
Posted by Mark Thomas <ma...@apache.org>.
On 05/08/2019 21:49, Chen Levy wrote:
> Hello Experts
>
> Several of my production servers were recently upgraded from Tomcat 9.0.14 to 9.0.21; immediately after the upgrade the servers started accumulating memory and open-files (on Linux) in a steady trend that was not observed before.
> After a couple of days (without reaching the memory or open-files limit and without throwing "OutOfMemoryError: Java heap space" or "IOException: Too many open files") the servers became unresponsive: any HTTPS request timed-out while HTTP requests continued to work correctly.
> Restarting the servers resolved the symptoms but the behavior persists and a restart is necessary every couple of days.
> I loaded a heap dump from an unresponsive server into MAT and received the following Leak Suspect:
>
> 105,871 instances of "org.apache.coyote.http2.Stream", loaded by "java.net.URLClassLoader..." occupy 7,581,549,904 (80.68%) bytes.
> These instances are referenced from one instance of "java.util.concurrent.ConcurrentHashMap$Node[]", loaded by "<system class loader>"
>
> The HashMap referenced in the report is "connections" inside ConnectionHandler.
> I suspect that these objects accumulate as clients may not close their connections correctly; regardless, I'd expect Tomcat to close the connections upon timeout.
> With keepAliveTimeout="20000" defined on UpgradeProtocol, I tested one simple HTTP2 connection's persistence on Chrome's net-internals.
> With 9.0.14 I can see the following at 20 seconds (as expected):
> ...
> t=7065701 [st= 64] HTTP2_SESSION_UPDATE_RECV_WINDOW
> --> delta = 6894
> --> window_size = 15728640
> t=7085708 [st=20071] HTTP2_SESSION_PING
> --> is_ack = false
> --> type = "received"
> --> unique_id = 2
> t=7085708 [st=20071] HTTP2_SESSION_PING
> --> is_ack = true
> --> type = "sent"
> --> unique_id = 2
> t=7085708 [st=20071] HTTP2_SESSION_CLOSE
> --> description = "Connection closed"
> --> net_error = -100 (ERR_CONNECTION_CLOSED)
> t=7085708 [st=20071] HTTP2_SESSION_POOL_REMOVE_SESSION
> t=7085708 [st=20071] -HTTP2_SESSION
>
> With 9.0.21 the connection does not close, even after several minutes.
> I believe the change in behavior stems the following commit: https://github.com/apache/tomcat/commit/c16d9d810a1f64cd768ff33058936cf8907e3117 and so I may be doing something wrong.
>
> Please let me know whether I have misconfigured, misunderstood, misdiagnosed, misbehaved or mis-something-else, and whether I should provide additional information
You haven't done anything wrong. This is a Tomcat bug. The bug is
related to the change you identified above. Based on my understanding at
this point, that change made an existing bug more obvious / more likely
to occur.
I am working on a fix which should be included in the next round of
releases which I expect to start in the next day or so.
Mark
>
> Current setup of the production servers:
> AdoptOpenJDK (build 11.0.3+7)
> Amazon Linux 2
>
> <Connector port="443" protocol="org.apache.coyote.http11.Http11NioProtocol"
> maxHttpHeaderSize="16384"
> maxThreads="500" minSpareThreads="25"
> enableLookups="false" disableUploadTimeout="true"
> connectionTimeout="10000"
> compression="on"
> SSLEnabled="true" scheme="https" secure="true">
> <UpgradeProtocol className="org.apache.coyote.http2.Http2Protocol"
> keepAliveTimeout="20000"/>
> <SSLHostConfig protocols="+TLSv1.2+TLSv1.3">
> <Certificate certificateKeystoreFile="tomcat.keystore"
> certificateKeyAlias="tomcat"
> certificateKeystorePassword=""
> certificateKeystoreType="PKCS12"/>
> </SSLHostConfig>
> </Connector>
>
> Thanks
> Chen
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org