You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Luke Meyer <lm...@vmware.com> on 2012/04/10 23:09:17 UTC
mod_jk - Firewall connection dropping
I have some questions about the documentation at http://tomcat.apache.org/connectors-doc/generic_howto/timeouts.html regarding the Firewall Connection Dropping section. I don't think it's quite right, but maybe I misunderstand.
To quote the sections in question:
" Many firewalls will allow connection closing, even if they dropped the connection for normal traffic. Therefore you should always use connection_pool_timeout and connection_pool_minsize on the JK side and connectionTimeout on the Tomcat side.
[...]
In case none of our recommendations help and you are definitively having problems with idle connection drops, you can disable the use of persistent connections when using JK together with Apache httpd. For this you set "JkOptions +DisableReuse" in your Apache httpd configuration. This will have a huge negative performance impact! "
The recommendation to use connection_pool_timeout seems good - close connections that have been idle so the firewall doesn't drop them. But I don't see what connectionTimeout on the AJP connector has to do with keepalive timeouts - it seems to be for initial connections only. I would think keepAliveTimeout is the way to get rid of connections that the firewall has dropped - yes?
I also don't understand why disabling reuse entirely is said to have a "huge negative performance impact!" The only impact I can see is that you have to reopen a connection for each request. If your servers are right next to each other on the network, or even on the same box, as is typical in most AJP setups, then connection time is negligible. In my tests (on the same box) the whole handshake took .1 millisecond - obviously, if there's some latency, that may be a few milliseconds, but in the scenario where a firewall is dropping inactive connections, I hardly think there would be so much throughput that this would matter. Certainly it's not going to matter compared to the risk of threads being tied up on dead connections. And if your network isn't perfect, but the latency is negligible, why not just disable reuse so you never have to take that risk?
And the biggest question... why does the default configuration assume that connections will always either stay open forever or be properly closed? By default, if connections are dropped, Tomcat eventually becomes unresponsive because the connector thread pool is exhausted, and you must restart. This looks really bad. Why not have a default keepAliveTimeout value that acknowledges the real world isn't perfect and frees threads from dead connections?
Re: mod_jk - Firewall connection dropping
Posted by Mark Thomas <ma...@apache.org>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 12/04/2012 16:20, Christopher Schultz wrote:
> On 4/12/12 4:37 AM, Mark Thomas wrote:
>> In short, disabling mod_jk keep-alive frequently has no
>> noticeable performance impact (some customers of mine reported a
>> performance improvement!)
>
> That sounds suspicious to me.
Not really. If cpong/cping isn't set up correctly and you are taking
Tomcat hosts up and down regularly (or the hosts are failing to
respond for other reasons) then without keep-alive disabled some
requests are going to get delayed by a timeout before httpd retries
another Tomcat instance. Those timeouts could impact the average
response time.
A correctly configured mod_jk setup should perform about the same with
or without keep-alive. The point is that without keep-alive the config
is usually far easier to get right.
Mark
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iQIcBAEBAgAGBQJPhwSrAAoJEBDAHFovYFnndpgP/jhHXv5H8SMl+uE+3Wppp/2Z
TQKG4HtqAtdcXi/yyYYro0t3BzknekUDVEX39qIFdDrBK1ogwZc6/bAZhrXGo+wx
dCwPdm1m34RpXGF5mWBYqVHdx/RvmgrKMM/Wt/P58nGL1NU7tOvbupPbtZ1CNig7
t/FCzMSxl7V0DzNHgtxlfOfWBNP8te2Eu9LsBxKNyZbUOEf6/yOoXZlGozIPdKKB
7/23FCnwHtAE2vLh0kvpTNlv0f6hj2Rgx3s/XX+Snc7LLbqRyzWOyvkEn0PrfV+5
Mgl4F+pcShd35CeKRD4+g0VdteJULdeT0LO816hRKDO7e/UNr/CVTh5xsXHVjwdK
59lVtACoNEysy/Tucm28ZEfIIosjRKvseEOhRhHZKWiAOXoL7++XYUkdixaiZ5ZX
c40FQ225H38emQb10wTEuhFCrqAZWFDlR+6W0ySL391lFXHuiPEGgekDRzNAm6Uh
Gy3uczA6q4NldnN2SvCr/8mx+iawkV+kLgsdIAluIKg/K0/yiT7+S6QUpAMgeyc5
z2RtLdfpMeqGW8mvf3QDa+z9ojMy3nUK7ukgB3ENBgOcpuJNSW1zYjXmRmMGJALr
FZ50aFhn+FLVwMd6OIggo/4faQ1hr4SrZTMlmnb6BNHMi0Oy3HK7yqV5g0pTKq8h
11WtLxb707rXbBd3kpAU
=Qnhw
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: mod_jk - Firewall connection dropping
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Mark,
On 4/12/12 4:37 AM, Mark Thomas wrote:
> HTTP keep-alive is nearly always between a client and a server
> across the internet. mod_jk keep-alive is nearly always between a
> client and a server on the same LAN and often on the same machine.
>
> HTTP keep-alive has no support for checking the liveness of the
> connection before it is used.
>
> mod_jk keep-alive supports cping/cpong and it is pretty much
> essential that it is used to ensure requests aren't sent to
> dropped connections or Tomcat instances taken out of service. The
> time taken for cping/cpong is very similar to that required to
> establish a new connection (and both are very, very small when
> compared to the time taken to process a request).
I have not done any performance testing, but I would suspect that TCP
connection setup is roughly equivalent to a CPING/CPONG. On the other
hand, if the environment demands something like stunnel, then creating
a new AJP connection also means an SSL negotiation, where a
CPING/CPONG is likely to be much faster.
> keep-alive does not scale for BIO connectors which require one
> thread per connection. Tomcat's BIO HTTP connector protects
> against thread exuastion due to keep-alive by disabling keep-alive
> once 75% of the threads are in use. Tomcat's BIO AJP has no
> exhaustion against thread exhaustion. Because mod_jk uses
> persistent connections folks often don't get that Tomcat maxThreads
> (on each instance) must equal the total number of httpd threads
> across all httpd instances that route requests to that Tomcat.
> Managing this corectly requires careful setting of timeouts and,
> despite Rainer and others writing some great docs, folks still
> frequently get it wrong.
That certainly is true: if one does not understand the network map
(including connection counts), then one will get weird connection
failures, etc.
But the maxThreads setting has little to do with persistent AJP
connections, except that (httpd) clients are less likely to be
starved-out of the app server if all connections are ephemeral.
> In short, disabling mod_jk keep-alive frequently has no noticeable
> performance impact (some customers of mine reported a performance
> improvement!)
That sounds suspicious to me.
> and greatly simplifies configuration - especially for those folks
> that don't have a complete grasp of all of the details.
I definitely agree with that.
> The notice in the docs about "huge negative performance impact" is
> a nonsense that needs to be removed. However, given the strength
> of feelings that were expressed when the diabling keep-alive
> feature was proposed (that lead to that notice in the docs) I am
> not prepared to remove it as I don't wish to start a commit war.
Fair enough.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk+G8qgACgkQ9CaO5/Lv0PAG4QCcDkJnlXx47DeD1rt4HiWCIhBm
Bj4AoIIpiw4+TlgR1xobK+E7C4dp4oQV
=f4Ga
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: mod_jk - Firewall connection dropping
Posted by Mark Thomas <ma...@apache.org>.
Christopher Schultz <ch...@christopherschultz.net> wrote:
>-----BEGIN PGP SIGNED MESSAGE-----
>Hash: SHA1
>
>Luke,
>
>On 4/11/12 5:17 PM, Luke Meyer wrote:
>> Just noticed what I was missing about connectionTimeout - it's the
>> default for keepAliveTimeout. So, that clears that up.
>>
>> Still don't think it's a "huge negative performance impact!" to
>> disable connection reuse.
>
>Well, if you think that HTTP keepalives represent a "huge positive
>performance impact" (and I certainly do), then the opposite ought to
>be true: if mod_jk has to re-establish contact with the backend Tomcat
>server for every single request, that will definitely have a negative
>performance impact.
Nope. You are not comparing apples and apples.
HTTP keep-alive is nearly always between a client and a server across the internet.
mod_jk keep-alive is nearly always between a client and a server on the same LAN and often on the same machine.
HTTP keep-alive has no support for checking the liveness of the connection before it is used.
mod_jk keep-alive supports cping/cpong and it is pretty much essential that it is used to ensure requests aren't sent to dropped connections or Tomcat instances taken out of service. The time taken for cping/cpong is very similar to that required to establish a new connection (and both are very, very small when compared to the time taken to process a request).
keep-alive does not scale for BIO connectors which require one thread per connection. Tomcat's BIO HTTP connector protects against thread exuastion due to keep-alive by disabling keep-alive once 75% of the threads are in use. Tomcat's BIO AJP has no exhaustion against thread exhaustion. Because mod_jk uses persistent connections folks often don't get that Tomcat maxThreads (on each instance) must equal the total number of httpd threads across all httpd instances that route requests to that Tomcat. Managing this corectly requires careful setting of timeouts and, despite Rainer and others writing some great docs, folks still frequently get it wrong.
In short, disabling mod_jk keep-alive frequently has no noticeable performance impact (some customers of mine reported a performance improvement!) and greatly simplifies configuration - especially for those folks that don't have a complete grasp of all of the details. The notice in the docs about "huge negative performance impact" is a nonsense that needs to be removed. However, given the strength of feelings that were expressed when the diabling keep-alive feature was proposed (that lead to that notice in the docs) I am not prepared to remove it as I don't wish to start a commit war. I'd be happy if the notice was toned down rather than removed but I'd rather see Rainer edit it to something he his happy with than I just remove it.
Mark
>
>- -chris
>-----BEGIN PGP SIGNATURE-----
>Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
>Comment: GPGTools - http://gpgtools.org
>Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
>iEYEARECAAYFAk+F/dkACgkQ9CaO5/Lv0PAVDwCcDm6/dCbAqgi0oaWESDUuRGmW
>sdgAn1i/q8hHIQcthwqriOZVJYZy2m1j
>=Boq3
>-----END PGP SIGNATURE-----
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: mod_jk - Firewall connection dropping
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Luke,
On 4/11/12 5:17 PM, Luke Meyer wrote:
> Just noticed what I was missing about connectionTimeout - it's the
> default for keepAliveTimeout. So, that clears that up.
>
> Still don't think it's a "huge negative performance impact!" to
> disable connection reuse.
Well, if you think that HTTP keepalives represent a "huge positive
performance impact" (and I certainly do), then the opposite ought to
be true: if mod_jk has to re-establish contact with the backend Tomcat
server for every single request, that will definitely have a negative
performance impact.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk+F/dkACgkQ9CaO5/Lv0PAVDwCcDm6/dCbAqgi0oaWESDUuRGmW
sdgAn1i/q8hHIQcthwqriOZVJYZy2m1j
=Boq3
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: mod_jk - Firewall connection dropping
Posted by Luke Meyer <lm...@vmware.com>.
Just noticed what I was missing about connectionTimeout - it's the default for keepAliveTimeout. So, that clears that up.
Still don't think it's a "huge negative performance impact!" to disable connection reuse.
----- Original Message -----
From: "Luke Meyer" <lm...@vmware.com>
To: users@tomcat.apache.org
Sent: Tuesday, April 10, 2012 5:09:17 PM
Subject: mod_jk - Firewall connection dropping
I have some questions about the documentation at http://tomcat.apache.org/connectors-doc/generic_howto/timeouts.html regarding the Firewall Connection Dropping section. I don't think it's quite right, but maybe I misunderstand.
To quote the sections in question:
" Many firewalls will allow connection closing, even if they dropped the connection for normal traffic. Therefore you should always use connection_pool_timeout and connection_pool_minsize on the JK side and connectionTimeout on the Tomcat side.
[...]
In case none of our recommendations help and you are definitively having problems with idle connection drops, you can disable the use of persistent connections when using JK together with Apache httpd. For this you set "JkOptions +DisableReuse" in your Apache httpd configuration. This will have a huge negative performance impact! "
The recommendation to use connection_pool_timeout seems good - close connections that have been idle so the firewall doesn't drop them. But I don't see what connectionTimeout on the AJP connector has to do with keepalive timeouts - it seems to be for initial connections only. I would think keepAliveTimeout is the way to get rid of connections that the firewall has dropped - yes?
I also don't understand why disabling reuse entirely is said to have a "huge negative performance impact!" The only impact I can see is that you have to reopen a connection for each request. If your servers are right next to each other on the network, or even on the same box, as is typical in most AJP setups, then connection time is negligible. In my tests (on the same box) the whole handshake took .1 millisecond - obviously, if there's some latency, that may be a few milliseconds, but in the scenario where a firewall is dropping inactive connections, I hardly think there would be so much throughput that this would matter. Certainly it's not going to matter compared to the risk of threads being tied up on dead connections. And if your network isn't perfect, but the latency is negligible, why not just disable reuse so you never have to take that risk?
And the biggest question... why does the default configuration assume that connections will always either stay open forever or be properly closed? By default, if connections are dropped, Tomcat eventually becomes unresponsive because the connector thread pool is exhausted, and you must restart. This looks really bad. Why not have a default keepAliveTimeout value that acknowledges the real world isn't perfect and frees threads from dead connections?
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org