You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Mike Wertheim <mw...@hyperreal.org> on 2011/12/15 19:49:20 UTC
keepalive issue
I have a Java app running on Tomcat 7.0.21 with APR 1.4.7. The app runs on
64-bit Java 7 Update 1 on CENT OS servers, with an A10 load balancer that
sends traffic to the servers. On average, each server is handling about 80
requests per second.
We are having problems with our network connections. In server.xml, if we
have keepalive on (maxKeepAliveRequests value greater than 1), then each
server has way too many connections in the ESTABLISHED state, and the load
balancer's NAT pool overflowsbecause it has too many connections to track.
If we have keepalive off (maxKeepAliveRequests value set to 1), then the
number of ESTABLISHED connections goes way down and the load balancer seems
ok, but the app starts refusing connections (we're not sure why).
Does this make sense to you? I'm not clear about why setting
maxKeepAliveRequests="1" would have such a drastic impact on the site's
performance (especially since most of our static files are on Akamai's CDN).
Here is what our Connectors in server.xml look like:
<Connector port="8080" protocol="HTTP/1.1" URIEncoding="utf-8"
connectionTimeout="5000" maxThreads="800"
maxKeepAliveRequests="1"
redirectPort="8443" />
<Connector port="8443" URIEncoding="utf-8"
connectionTimeout="5000" maxThreads="800"
maxKeepAliveRequests="1"
SSLCertificateFile="/home/user/ssl/2010.www.foo.com.crt"
SSLCertificateKeyFile="/home/user/ssl/www.foo.com.key"
SSLCertificateChainFile="/home/user/ssl/intermediateCA.cer"
scheme="https" secure="true" SSLEnabled="true" clientAuth="false"
sslProtocol="TLS"/>
Can anyone give me any insight on what the problem might be and how to
troubleshoot this?
RE: keepalive issue
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: mike.wertheim@gmail.com [mailto:mike.wertheim@gmail.com] On Behalf Of Mike Wertheim
> Subject: Re: keepalive issue
> My question was basically: If I'm using APR, can I assume that I'll
> get the APR Connector
Yes, you'll have the APR <Connector>, with all of its attributes.
> Also, I reduced the ConnectionTimeout from the 20-second default
> down to 5 seconds, and the app seems to work fine. Would it be
> advisable to reduce it even further?
Not really something that can be answered here. If you're seeing that it makes a difference, then it sounds like you have either balky clients or network problems. It's very unusual for a TCP_OPEN to show up without real traffic right on its tail - unless somebody's trying to take your site down by brute force
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: keepalive issue
Posted by André Warnier <aw...@ice-sa.com>.
Mike Wertheim wrote:
> APR is definitely being loaded. The DLL is there, and the log says it's
> being used.
>
> My question was basically: If I'm using APR, can I assume that I'll get the
> APR Connector, or do I need to explicitly specify that I want the APR
> Connector? (I'm assuming the answer is 'yes' to the first question and
> 'no' the the second question.)
>
> Also, I reduced the ConnectionTimeout from the 20-second default down to 5
> seconds, and the app seems to work fine. Would it be advisable to reduce
> it even further? Assuming that all of my app's users are on a decent DSL
> connection or cable connection, could I get away with reducing it to
> perhaps to 2 seconds, or just 1 second?
connectionTimeout
The number of milliseconds this Connector will wait, after accepting a connection, for the
request URI line to be presented. Use a value of -1 to indicate no (i.e. infinite)
timeout. The default value is 60000 (i.e. 60 seconds) but note that the standard
server.xml that ships with Tomcat sets this to 20000 (i.e. 20 seconds).
keepAliveTimeout
The number of milliseconds this Connector will wait for another HTTP request before
closing the connection. The default value is to use the value that has been set for the
connectionTimeout attribute. Use a value of -1 to indicate no (i.e. infinite) timeout.
They are two different things. It just happens that the default value of the
keepAliveTimeout is the value of the connectionTimeout (it would not make much sense
otherwise).
The connectionTimeout comes into play in this scenario :
- a client establishes a TCP connection to your server, and then it does not send a
request, but just maintains that connection and waits.
Ever heard of DOS attacks ? This is a good way..
Personally, I think that the standard setting of 20 seconds is still quite high.
I cannot imagine why a client would set up a connection to a server, and then not send a
request for the next 20 seconds, unless it is a malicious client.
(Of course, some clients and some lines are slow, so don't reduce it beyond a reasonable
value).
But the keepAliveTimeout is the one I was talking about.
It comes into play *after* the response to each request has been sent. The server keeps
the connection open, in the hope that another request from the client would arrive on it,
before it has to throw away this connection and start all over again for the next request.
But if you know that in your applications, such a "follow-up" request can never come later
than 2 seconds after the first response, then reduce it, maybe to 3-4 seconds, to take
into account small network delays.
The point is : you want to avoid having to build up and tear down SSL connections more
than necessary; but you do not want connections and threads sitting doing nothing for too
long either. So this setting should be a good compromise, /in your situation/.
Take into account that your situation may change over time, so don't set things too sharp.
Change one setting at a time, a little bit at a time, and observe the result.
Many of these settings have indirect effects on others, so it can quickly become confusing
if you change several things at the same time.
And if there was one set of settings which performed best in all cases, then the people
who make and package Tomcat would have set those already in the default configuration (or
they would be fixed in the code).
>
>
> On Fri, Dec 16, 2011 at 8:20 AM, Caldarale, Charles R <
> Chuck.Caldarale@unisys.com> wrote:
>
>>> From: mike.wertheim@gmail.com [mailto:mike.wertheim@gmail.com] On
>> Behalf Of Mike Wertheim
>>> Subject: Re: keepalive issue
>>> In my server.xml, I have this line which turns on APR:
>> "Enables" would be a better word; it does not force the usage of APR.
>>
>>> With this configuration, can I assume that the Connector that I'm using
>> is
>>> the APR/native Connector (the 3rd one in the Connector Comparison table)?
>> No; APR will be used only if the .so (or .dll) can be found and loaded.
>> The Tomcat logs should tell you whether or not that happened.
>>
>> - Chuck
>>
>>
>> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
>> MATERIAL and is thus for use only by the intended recipient. If you
>> received this in error, please contact the sender and delete the e-mail and
>> its attachments from all computers.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: keepalive issue
Posted by Mike Wertheim <mw...@hyperreal.org>.
APR is definitely being loaded. The DLL is there, and the log says it's
being used.
My question was basically: If I'm using APR, can I assume that I'll get the
APR Connector, or do I need to explicitly specify that I want the APR
Connector? (I'm assuming the answer is 'yes' to the first question and
'no' the the second question.)
Also, I reduced the ConnectionTimeout from the 20-second default down to 5
seconds, and the app seems to work fine. Would it be advisable to reduce
it even further? Assuming that all of my app's users are on a decent DSL
connection or cable connection, could I get away with reducing it to
perhaps to 2 seconds, or just 1 second?
On Fri, Dec 16, 2011 at 8:20 AM, Caldarale, Charles R <
Chuck.Caldarale@unisys.com> wrote:
> > From: mike.wertheim@gmail.com [mailto:mike.wertheim@gmail.com] On
> Behalf Of Mike Wertheim
> > Subject: Re: keepalive issue
>
> > In my server.xml, I have this line which turns on APR:
>
> "Enables" would be a better word; it does not force the usage of APR.
>
> > With this configuration, can I assume that the Connector that I'm using
> is
> > the APR/native Connector (the 3rd one in the Connector Comparison table)?
>
> No; APR will be used only if the .so (or .dll) can be found and loaded.
> The Tomcat logs should tell you whether or not that happened.
>
> - Chuck
>
>
> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
> MATERIAL and is thus for use only by the intended recipient. If you
> received this in error, please contact the sender and delete the e-mail and
> its attachments from all computers.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
RE: keepalive issue
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: mike.wertheim@gmail.com [mailto:mike.wertheim@gmail.com] On Behalf Of Mike Wertheim
> Subject: Re: keepalive issue
> In my server.xml, I have this line which turns on APR:
"Enables" would be a better word; it does not force the usage of APR.
> With this configuration, can I assume that the Connector that I'm using is
> the APR/native Connector (the 3rd one in the Connector Comparison table)?
No; APR will be used only if the .so (or .dll) can be found and loaded. The Tomcat logs should tell you whether or not that happened.
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: keepalive issue
Posted by Mike Wertheim <mw...@hyperreal.org>.
In my server.xml, I have this line which turns on APR:
<Listener className="org.apache.catalina.core.AprLifecycleListener"
SSLEngine="on" />
And the Connectors look like this:
<Connector port="8080" protocol="HTTP/1.1" URIEncoding="utf-8"
connectionTimeout="5000" maxThreads="800"
maxKeepAliveRequests="1"
redirectPort="8443" />
<Connector port="8443" URIEncoding="utf-8"
connectionTimeout="5000" maxThreads="800"
maxKeepAliveRequests="1"
SSLCertificateFile="/home/user/ssl/2010.www.foo.com.crt"
SSLCertificateKeyFile="/home/user/ssl/www.foo.com.key"
SSLCertificateChainFile="/home/user/ssl/intermediateCA.cer"
scheme="https" secure="true" SSLEnabled="true" clientAuth="false"
sslProtocol="TLS"/>
With this configuration, can I assume that the Connector that I'm using is
the APR/native Connector (the 3rd one in the Connector Comparison table)?
On Fri, Dec 16, 2011 at 5:21 AM, Caldarale, Charles R <
Chuck.Caldarale@unisys.com> wrote:
> > From: André Warnier [mailto:aw@ice-sa.com]
> > Subject: Re: keepalive issue
>
> > Since you are under Tomcat 7, you may want to have a look at
> > the Executor feature.
>
> Which has been available since at least Tomcat 6 (I'm too lazy to look at
> the older docs).
>
> > Again, I am not an expert, but if I understand correctly what
> > this does, it avoids these "keep-alive" threads sitting around
> > doing nothing, and allows them to be used to service requests
> > on other connections in the meantime.
>
> No, an <Executor> does nothing of the sort; rather it shares a thread pool
> across multiple <Connector>s.
>
> If you want to avoid threads attached to connections, use a <Connector>
> other than the basic one:
> http://tomcat.apache.org/tomcat-7.0-doc/config/http.html
>
> Look at the table at the bottom of the page for a comparison of
> <Connector> types. The "Wait for next request" row is the one of interest
> here.
>
> - Chuck
>
>
> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
> MATERIAL and is thus for use only by the intended recipient. If you
> received this in error, please contact the sender and delete the e-mail and
> its attachments from all computers.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
RE: keepalive issue
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: André Warnier [mailto:aw@ice-sa.com]
> Subject: Re: keepalive issue
> Since you are under Tomcat 7, you may want to have a look at
> the Executor feature.
Which has been available since at least Tomcat 6 (I'm too lazy to look at the older docs).
> Again, I am not an expert, but if I understand correctly what
> this does, it avoids these "keep-alive" threads sitting around
> doing nothing, and allows them to be used to service requests
> on other connections in the meantime.
No, an <Executor> does nothing of the sort; rather it shares a thread pool across multiple <Connector>s.
If you want to avoid threads attached to connections, use a <Connector> other than the basic one:
http://tomcat.apache.org/tomcat-7.0-doc/config/http.html
Look at the table at the bottom of the page for a comparison of <Connector> types. The "Wait for next request" row is the one of interest here.
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: keepalive issue
Posted by André Warnier <aw...@ice-sa.com>.
Mike Wertheim wrote:
> I have a Java app running on Tomcat 7.0.21 with APR 1.4.7. The app runs on
> 64-bit Java 7 Update 1 on CENT OS servers, with an A10 load balancer that
> sends traffic to the servers. On average, each server is handling about 80
> requests per second.
>
> We are having problems with our network connections. In server.xml, if we
> have keepalive on (maxKeepAliveRequests value greater than 1), then each
> server has way too many connections in the ESTABLISHED state, and the load
> balancer's NAT pool overflowsbecause it has too many connections to track.
> If we have keepalive off (maxKeepAliveRequests value set to 1), then the
> number of ESTABLISHED connections goes way down and the load balancer seems
> ok, but the app starts refusing connections (we're not sure why).
>
> Does this make sense to you? I'm not clear about why setting
> maxKeepAliveRequests="1" would have such a drastic impact on the site's
> performance (especially since most of our static files are on Akamai's CDN).
>
> Here is what our Connectors in server.xml look like:
>
> <Connector port="8080" protocol="HTTP/1.1" URIEncoding="utf-8"
> connectionTimeout="5000" maxThreads="800"
> maxKeepAliveRequests="1"
> redirectPort="8443" />
>
> <Connector port="8443" URIEncoding="utf-8"
> connectionTimeout="5000" maxThreads="800"
> maxKeepAliveRequests="1"
> SSLCertificateFile="/home/user/ssl/2010.www.foo.com.crt"
> SSLCertificateKeyFile="/home/user/ssl/www.foo.com.key"
> SSLCertificateChainFile="/home/user/ssl/intermediateCA.cer"
> scheme="https" secure="true" SSLEnabled="true" clientAuth="false"
> sslProtocol="TLS"/>
>
> Can anyone give me any insight on what the problem might be and how to
> troubleshoot this?
>
Hi.
What I'll say below is not coming from any deep knowledge of the Tomcat code, rather from
some reasonable knowledge of HTTP in general. Take it with the appropriate grain of salt.
Setting up an SSL connection is not trivial, compared to a normal TCP connection.
When you set mexKeepAliveRequeets to 1, you essentially force a new SSL negociation at
each request, which seems quite inefficient.
On the other hand, when using the normal Tomcat <Connector> logic (as you seem to be doing
above), if KeepAlive is enabled, it means that one Tomcat thread will be dedicated to an
incoming connection, to process the first request, and then it will "hang around" during a
certain time, in case there are more requests coming over the same connection.
If there are more requests in quick succession, it will process them, and this is
efficient since no new SSL connection has to be built.
If there are no further requests after the first one however, then the thread will just
sit around there waiting on that connection, until the KeepAlive timeout kicks in, the
connection is closed, and the thread can return to the pool, to do something useful.
So if most of your client interactions consist of just one request, this may turn out to
be rather inefficient, since you are blocking connections and threads to just sit around
doing nothing, for the duration of the KeepAlive timeout.
If in addition this timeout is set on the high side (more than 2-3 seconds), you make it
even worse, because that's the time the thread will sit around doing nothing.
KeepAlive is particularly effective when a client makes a request for an initial page and
gets it, and in that page are lots of elements which require further requests before the
page can be displayed (think of a page with lots of thumbnails e.g.).
But in your case, you seem to say that many of these "secondary" elements might be on
another server, so these secondary requests would never go to Tomcat, and it might well be
the case that the Tomcat threads spend most of their time waiting for the keep-alive
timeout to expire.
Since you are under Tomcat 7, you may want to have a look at the Executor feature.
http://tomcat.apache.org/tomcat-7.0-doc/config/executor.html
Again, I am not an expert, but if I understand correctly what this does, it avoids these
"keep-alive" threads sitting around doing nothing, and allows them to be used to service
requests on other connections in the meantime.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org