You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@tomcat.apache.org by Mason Meier <ma...@gmail.com> on 2019/01/08 23:51:15 UTC

Tomcat memory growth while using TLS

Hello,

I'm running Tomcat-8.5 with TLS and I've noticed substantial memory growth
with requests over time, to the point that if I run Tomcat in Docker and
make constant requests to it, Docker will kill the container due to
excessive memory utilization. The problem occurs with standalone Tomcat as
well. Over the course of millions of requests, the memory usage of the
Tomcat process grows continuously, seemingly without bound.

I've done a fair amount of testing on AWS EC2 instances and some local
machines, and here are my observations:
  * 'org.apache.tomcat.util.net.openssl.OpenSSLImplementation' seems to
increase memory utilization more quickly and consistently than
'org.apache.tomcat.util.net.jsse.JSSEImplementation'. The
JSSEImplementation doesn't cause the memory to grow in certain setups.
  * Limiting the heap size of the JVM does not affect the amount of memory
consumed
  * Other than decreasing a small amount (presumably due to garbage
collection), memory utilization stays consistent after the flow of requests
is stopped

My testing consists of:
1. Installing brand new versions of everything
2. Generating a certificate
3. Changing my server.xml to the one shown below
4. Replacing 'webapps/ROOT/index.jsp' with a shorter payload
5. Starting Tomcat
6. Making millions of parallel requests from a different computer and
watching Tomcat's memory utilization grow

Is there anything in my 'server.xml' that would be cause for concern? Are
there any known memory leaks in Tomcat's OpenSSL implementation? What steps
can I take to debug this problem?

Versions:
Tomcat - apache-tomcat-8.5.37
Java - JDK-1.8u191
OpenSSL - openssl-1.0.2q
APR - apr-1.6.5
Tomcat Native - tomcat-native-1.2.19
OS - Amazon Linux release 2 (Karoo)
uname -a - Linux 4.14.77-81.59.amzn2.x86_64 #1 SMP Mon Nov 12 21:32:48 UTC
2018 x86_64 x86_64 x86_64 GNU/Linux

Here is my 'server.xml' file:
<?xml version="1.0" encoding="UTF-8"?>
<Server port="8005" shutdown="SHUTDOWN">
  <Listener className="org.apache.catalina.startup.VersionLoggerListener" />
  <Listener className="org.apache.catalina.core.AprLifecycleListener"
SSLEngine="on" />
  <Listener
className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />
  <Listener
className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />
  <Listener
className="org.apache.catalina.core.ThreadLocalLeakPreventionListener" />

  <Service name="Catalina">
    <Connector protocol="org.apache.coyote.http11.Http11NioProtocol"

 sslImplementationName="org.apache.tomcat.util.net.openssl.OpenSSLImplementation"
           port="8443"
           SSLEnabled="true" scheme="https" secure="true"
           keystoreFile="server.keystore" keystorePass="<REDACTED>"
           clientAuth="optional" sslProtocol="TLS"/>
    <Engine name="Catalina" defaultHost="localhost">
      <Host name="localhost"  appBase="webapps"
            unpackWARs="true" autoDeploy="true">
        <Valve className="org.apache.catalina.valves.AccessLogValve"
directory="logs"
               prefix="localhost_access_log" suffix=".txt"
               pattern="%h %l %u %t &quot;%r&quot; %s %b" />
      </Host>
    </Engine>
  </Service>
</Server>

Thanks,
Mason

Re: Tomcat memory growth while using TLS

Posted by "info@flyingfischer.ch" <in...@flyingfischer.ch>.

Am 09.01.19 um 11:14 schrieb Mark Thomas:
> On 08/01/2019 23:51, Mason Meier wrote:
>> Hello,
>>
>> I'm running Tomcat-8.5 with TLS and I've noticed substantial memory growth
>> with requests over time, to the point that if I run Tomcat in Docker and
>> make constant requests to it, Docker will kill the container due to
>> excessive memory utilization. The problem occurs with standalone Tomcat as
>> well. Over the course of millions of requests, the memory usage of the
>> Tomcat process grows continuously, seemingly without bound.
> I've behaviour like this in the past. From memory there was some caching
> in the TLS implementation at the root of most of it that could be
> controlled with some system properties.
>
> It may be you are seeing the same thing. Or you may have found a memory
> leak. The next step would be to use a profiler to see where the memory
> is being used.
>
>> I've done a fair amount of testing on AWS EC2 instances and some local
>> machines, and here are my observations:
>>   * 'org.apache.tomcat.util.net.openssl.OpenSSLImplementation' seems to
>> increase memory utilization more quickly and consistently than
>> 'org.apache.tomcat.util.net.jsse.JSSEImplementation'. The
>> JSSEImplementation doesn't cause the memory to grow in certain setups.
> Can you share some configs that demonstrate an issue and some that
> don't. That might help narrow down what is going on.
>
> Mark
>

We do also see frequent OOM memories with the latest Tomcat versions.
Free memory shrinks to a critical value. With previous versions this
behaviour was not here. However, we use a limited machine and I need to
investigate further.

Markus

>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

Re: Tomcat memory growth while using TLS

Posted by John Dale <jc...@gmail.com>.

I love it when a memory leak bites the dust .. it's a great feeling to
extend process life.

Fixed one yesterday in a capture/encode buffer of my mp3 streamer.

Kudos, Mark!

:)

John


On 1/12/19, info@flyingfischer.ch <in...@flyingfischer.ch> wrote:
> Am 11.01.19 um 18:23 schrieb Mark Thomas:
>> Found it.
>>
>> The leak impacted NIO and NIO2 when used with OpenSSL.
>>
>> The bug is in Tomcat Native. I have a fix that I am currently testing.
>> That fix should be in the next Tomcat Native release.
>>
>> For those interested in the technical details, Tomcat Native allocates
>> some Native memory to track various attributes of each connection.
>>
>> In APR/Native this memory is allocated from a new APR memory pool that
>> is associated with the connection. When the connection closes, the pool
>> is destroyed and the memory freed. All is good.
>>
>> In NIO[2]+OpenSSL the per connection memory was allocated from the APR
>> memory pool associated with the Connector so the memory was not released
>> until the Connector was stopped. The fix is to create a pool per
>> connection and use that for the per connection allocations. With a fix
>> to do that in place, memory use is now broadly static under extended TLS
>> load.
>>
>> Mark
>>
>
> Thanks Mark!
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org