You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@httpd.apache.org by Darren Kukulka <Da...@connaught.plc.uk> on 2009/10/29 10:44:51 UTC
[users@httpd] Apache Close-Wait ports issue
Hi All
Got an interesting problem which has occurred several times now..
Painting a brief picture ;
Several applications running on Tomcat 6.0.19 (64-bit) - several cluster
groups
Backend SQL Server 2005 clustered DBs (several)
Front end Apache Web servers (2.2.9 with mod_jk 1.2.19), load balanced
via secure network appliance
Basically, we have an issue with one of the DB clusters which stops
servicing requests from the app servers
The app servers can continue to connect to remaining DB clusters OK.
The problem is that all the web servers stop responding to external
requests. Looking at the ports in use on each server, there are a
number of established sessions between Apache and Tomcat, which seems
OK. All of the web servers also have exactly 30 close-wait sessions
with the secure network appliance. We think this is what is stopping
any requests from being services by the web servers. However, even
trying to browse locally on each server to the root just times out.
We do not enforce keepalive or timeouts for TCP/HTTP, and we use
practically all the defaults for these directives. All web servers are
configures identically.
I can't find any reference to a directive that may limit connections to
30.
Restarting the web servers allows users to get to applications using the
good databases. We have already upgraded one of the load balanced web
servers to Apache 2.2.14 with mod_jk 1.2.28, and we intend to do the
rest of the next few days. Going through the changelogs there are one
or two items that might have a bearing on the issue but nothing obvious.
Does anybody have any advice on this?
Cheers,
Darren.
Connaught plc is a FTSE 250 company. We are the UK's leading provider of integrated services operating in the compliance, social housing and public sector markets.
Please visit our website to see a full list of Connaught's Registered Companies www.connaught.plc.uk/group/aboutconnaught/registeredcompanies
Disclaimer:
The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete this message. Connaught plc, Head Office 01392 444546
Re: [users@httpd] Apache Close-Wait ports issue
Posted by André Warnier <aw...@ice-sa.com>.
Darren Kukulka wrote:
>
> So as we're not specifying any directives in the httpd.conf, with respect to connection control, the default of KeepAlive=on and KeepAliveTimout=15 applies? Would this not clear out the close-wait connections?
>
No.
KeepAliveTimeout 15
is *very* long nowadays. It means that, after a browser has sent its
last request (and is maybe never going to make another request), the
Apache server's child/thread which was handling that connection, will
wait for up to 15 seconds, doing nothing, before it decides that the
browser is never going to ask for something else, and then close its
side of the connection and, possibly, signal that it is now available to
handle another request.
Try setting to 3 for instance.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
RE: [users@httpd] Apache Close-Wait ports issue
Posted by Darren Kukulka <Da...@connaught.plc.uk>.
Thanks for the reply Andre...responses in line
> -----Original Message-----
> From: André Warnier [mailto:aw@ice-sa.com]
> Sent: 29 October 2009 10:47
> To: users@httpd.apache.org
> Subject: Re: [users@httpd] Apache Close-Wait ports issue
>
> Welcome ;-)
>
> Darren Kukulka wrote:
> >
> > Painting a brief picture ;
> > Several applications running on Tomcat 6.0.19 (64-bit) - several
> cluster
> > groups
> > Backend SQL Server 2005 clustered DBs (several)
> > Front end Apache Web servers (2.2.9 with mod_jk 1.2.19), load
> balanced
> > via secure network appliance
> >
> ...
> > The problem is that all the web servers stop responding to external
> > requests. Looking at the ports in use on each server...
> . All of the web servers also have exactly 30 close-wait sessions
> > with the secure network appliance.
>
>
> If I get this right, you have
>
> browsers <--> network appliance <--> Apache httpd servers <--> other
> stuff
> and your CLOSE_WAIT sockets are on the Apache httpd servers, for
> connections with the network appliance, right ?
>
Correct - the connections from the Web Servers to the Tomcat app servers down the line remain established.
> Did you check (with "netstat -pan" e.g.) if that is really the case ?
>
> If that is correct, it would mean that Apache has closed these sockets,
> but is waiting for the network appliance to close it's side of these
> connections, which never happens. Any idea why that might be ?
>
> Another question : when that happens, how many Apache children/threads
> are running ?
> Maybe you just have 30 Apache children/threads that are "stuck" because
> they are waiting for the network appliance to close these sockets, and
> it never happens.
> As long as they would be waiting for that, they would never be recycled
> by Apache to handle new requests, which may explain what you're seeing
> ?
>
So as we're not specifying any directives in the httpd.conf, with respect to connection control, the default of KeepAlive=on and KeepAliveTimout=15 applies? Would this not clear out the close-wait connections?
We've queried the network appliance vendor but they are unaware of any reasons the appliances would not close the sockets.
>
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server
> Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
Connaught plc is a FTSE 250 company. We are the UK's leading provider of integrated services operating in the compliance, social housing and public sector markets.
Please visit our website to see a full list of Connaught's Registered Companies www.connaught.plc.uk/group/aboutconnaught/registeredcompanies
Disclaimer:
The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete this message. Connaught plc, Head Office 01392 444546
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Apache Close-Wait ports issue
Posted by André Warnier <aw...@ice-sa.com>.
Welcome ;-)
Darren Kukulka wrote:
>
> Painting a brief picture ;
> Several applications running on Tomcat 6.0.19 (64-bit) - several cluster
> groups
> Backend SQL Server 2005 clustered DBs (several)
> Front end Apache Web servers (2.2.9 with mod_jk 1.2.19), load balanced
> via secure network appliance
>
...
> The problem is that all the web servers stop responding to external
> requests. Looking at the ports in use on each server...
. All of the web servers also have exactly 30 close-wait sessions
> with the secure network appliance.
If I get this right, you have
browsers <--> network appliance <--> Apache httpd servers <--> other
stuff
and your CLOSE_WAIT sockets are on the Apache httpd servers, for
connections with the network appliance, right ?
Did you check (with "netstat -pan" e.g.) if that is really the case ?
If that is correct, it would mean that Apache has closed these sockets,
but is waiting for the network appliance to close it's side of these
connections, which never happens. Any idea why that might be ?
Another question : when that happens, how many Apache children/threads
are running ?
Maybe you just have 30 Apache children/threads that are "stuck" because
they are waiting for the network appliance to close these sockets, and
it never happens.
As long as they would be waiting for that, they would never be recycled
by Apache to handle new requests, which may explain what you're seeing ?
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org