You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@httpd.apache.org by pt...@intec.ugent.be on 2008/02/25 16:37:11 UTC
[users@httpd] Server hanging with requests in "W" state
Hello,
We're having some strange problems with our webservers (https), which are
Apache 2.0.52 on RHEL4 and 2.0.46 on RHEL 3.
The problem is that, under a certain load or after a while, the number of
connections to one of the webservers stuck in the "W" state (as indicate
by server-status?notable) steadily starts increasing. The "SS" value
listed for each request on this page is very large (i.e. >> any apache
timeout setting)
Having searched the archives, I have found threads on DoS attacks
(resulting in lots of connections in the "reading" state with little more
information).
The number of connections from any single ip, however, is at most 20-30 at
the point where "MaxClients" connections have frozen into the "W" state
(512 W's in our case). According to netstat, most of the connections to
the https server are in state "ESTABLISHED" (about half), then a lot of
the connections are "CLOSE_WAIT" and the remainder is in state "SYN_RECV".
The Apache servers connect to Websphere 5, WebSphere 6 (Websphere plugin)
and Tomcat/JBoss (mod_jk) application servers and use the CA/Netegrity
Siteminder plugin.
Bascially, my questions are:
1. When, exactly, is a connection listed in "W(riting)" state? Does this
mean the http server has already received the necessary responses from
backend application servers (if necessary)? Or can this state occur when
the http server is waiting for some backend server?
I'm looking to determine if maybe one of the backend plugins has not been
set correctly (assuming these plugins correctly honour timeout values
presented to them).
2. Has anyone experienced this kind of server behaviour, and if so, could
share some thoughts on possible (trivial and non-trivial) causes,
testcaeses and solutions?
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Server hanging with requests in "W" state
Posted by Joshua Slive <jo...@slive.ca>.
On Wed, Feb 27, 2008 at 3:43 AM, <pt...@intec.ugent.be> wrote:
>
> > Whats your timeout and keepalivetimeout settings set to ?
> >
> > If you have them more than 15 secs ? then you'll DOS yourself
> >
> > IE will hold open connections to the server for as long as it can...
> >
>
>
> Timeout 300
> KeepAliveTimeout 15
>
> SSl config contains directives to deal with Internet Explorer SSL
> keepalive issues as in previous post.
>
> I've been thinking of loading additional logging modules to debug this
> problem.
>
> mod_log_forensic seems unnecessary: it only helps to check which
> connections are hanging it seems, and I have no problem determining that
> by looking at my server-status page.
>
> Since these requests are hanging, they don't show up in the apache log
> file, so I would need some module to log every bit of traffic as soon as
> it arrives or is generated - maybe mod_dump_io will allow me to track
> traffic over a connection as a function of time?
mod_log_forensic does exactly what you want. It logs immediately on
receiving the connection, and if you kill the server when there are
hung connections and then run the check_forensic script, you'll get a
list of all the in-process connections.
Joshua.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Server hanging with requests in "W" state
Posted by pt...@intec.ugent.be.
> Whats your timeout and keepalivetimeout settings set to ?
>
> If you have them more than 15 secs ? then you'll DOS yourself
>
> IE will hold open connections to the server for as long as it can...
>
Timeout 300
KeepAliveTimeout 15
SSl config contains directives to deal with Internet Explorer SSL
keepalive issues as in previous post.
I've been thinking of loading additional logging modules to debug this
problem.
mod_log_forensic seems unnecessary: it only helps to check which
connections are hanging it seems, and I have no problem determining that
by looking at my server-status page.
Since these requests are hanging, they don't show up in the apache log
file, so I would need some module to log every bit of traffic as soon as
it arrives or is generated - maybe mod_dump_io will allow me to track
traffic over a connection as a function of time?
Also, after reading the "exact" definition of the Timeout directive, it's
still not very clear to me what will happen if a client requests a URL
which requires response from an application server and this application
server takes longer than 300s to respond? (I suppose this shouldn't happen
if all application server plugins have timeout settings > 300s)
Will apache terminate some connection (with the client? what with the
"dependent" connection to the application server?)?
Thanks,
Pieter
> On Tue, 2008-02-26 at 09:29 +0100, pthyseba@intec.ugent.be wrote:
>> Hello,
>>
>> > On 25/02/2008, pthyseba@intec.ugent.be <pt...@intec.ugent.be>
>> wrote:
>> >> We're having some strange problems with our webservers (https),
>> which
>> >> are
>> >> Apache 2.0.52 on RHEL4 and 2.0.46 on RHEL 3.
>> >
>> > This could be due to MSIE's duff SSL implementation. Do you have
>> > something like this in your SSL config?
>> >
>> > BrowserMatch ".*MSIE.*" \
>> > nokeepalive ssl-unclean-shutdown \
>> > downgrade-1.0 force-response-1.0
>> >
>> >
>>
>>
>> We do have this kind of line in our SSL configuration. However, we have
>> also seen this problem (512 W's with increasingly high "SS" values in
>> server-status, until even server-status can no longer be requested) on
>> our
>> non-ssl web servers, albeit a lot less frequently (possibly because of
>> lower load triggering our scenario less frequently).
>>
>> I'm wondering what the technical (apache, sockets, OS,...) reason is for
>> the absence of some "Apache kill switch" which would terminate (that is,
>> free the apache slot and make it available to another client) a
>> connection
>> with an "SS" value greater than some threshold - or is this supposed to
>> be
>> solved by any of the apache Timeout settings?
>>
>> If not, I'll have to dig further into the plugins we're running.
>>
>> Is there a good overview available which explains the relationship
>> between
>> Apache timeout settings and e.g. worker.properties (mod_jk) timeout
>> parameters (e.g. parameter1 >= parameter2) ? The documentation of each
>> package seems to only explain its own configuration directives, but I'm
>> thinking our problems may be related to some not-so-good combination of
>> timeout settings (at the moment they all have values > 0 and < infinite
>> though).
>>
>> Thx,
>>
>> Pieter
>>
>>
>> > noodl
>> >
>> > ---------------------------------------------------------------------
>> > The official User-To-User support forum of the Apache HTTP Server
>> Project.
>> > See <URL:http://httpd.apache.org/userslist.html> for more info.
>> > To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
>> > " from the digest: users-digest-unsubscribe@httpd.apache.org
>> > For additional commands, e-mail: users-help@httpd.apache.org
>> >
>>
>> ---------------------------------------------------------------------
>> The official User-To-User support forum of the Apache HTTP Server
>> Project.
>> See <URL:http://httpd.apache.org/userslist.html> for more info.
>> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
>> " from the digest: users-digest-unsubscribe@httpd.apache.org
>> For additional commands, e-mail: users-help@httpd.apache.org
>>
>
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
RE: [users@httpd] Server hanging with requests in "W" state
Posted by Narendra Verma <na...@impetus.co.in>.
Hi,
Thanks for replying to me.
As you told about time out I configured it as follows:
worker.worker1.socket_keepalive=1
worker.worker1.socket_timeout=300
Like this, But still my problem is not solved.
What I require specially:
Apache --- mod_jk connector ---tomcat
Caching by mod_disk_cache and mod_cache
Now if I changed any static file at tomcat(that is already cached at apache)
must be re-cached in further request without shutting down apache and
tomcat.
Is it possible ? if so please can you give me guide me for configuration the
same?
Thanks a lot..
Narendra
-----Original Message-----
From: David Cassidy [mailto:david@twocats.co.uk]
Sent: Tuesday, February 26, 2008 7:20 PM
To: users@httpd.apache.org
Subject: Re: [users@httpd] Server hanging with requests in "W" state
Whats your timeout and keepalivetimeout settings set to ?
If you have them more than 15 secs ? then you'll DOS yourself
IE will hold open connections to the server for as long as it can...
On Tue, 2008-02-26 at 09:29 +0100, pthyseba@intec.ugent.be wrote:
> Hello,
>
> > On 25/02/2008, pthyseba@intec.ugent.be <pt...@intec.ugent.be> wrote:
> >> We're having some strange problems with our webservers (https), which
> >> are
> >> Apache 2.0.52 on RHEL4 and 2.0.46 on RHEL 3.
> >
> > This could be due to MSIE's duff SSL implementation. Do you have
> > something like this in your SSL config?
> >
> > BrowserMatch ".*MSIE.*" \
> > nokeepalive ssl-unclean-shutdown \
> > downgrade-1.0 force-response-1.0
> >
> >
>
>
> We do have this kind of line in our SSL configuration. However, we have
> also seen this problem (512 W's with increasingly high "SS" values in
> server-status, until even server-status can no longer be requested) on our
> non-ssl web servers, albeit a lot less frequently (possibly because of
> lower load triggering our scenario less frequently).
>
> I'm wondering what the technical (apache, sockets, OS,...) reason is for
> the absence of some "Apache kill switch" which would terminate (that is,
> free the apache slot and make it available to another client) a connection
> with an "SS" value greater than some threshold - or is this supposed to be
> solved by any of the apache Timeout settings?
>
> If not, I'll have to dig further into the plugins we're running.
>
> Is there a good overview available which explains the relationship between
> Apache timeout settings and e.g. worker.properties (mod_jk) timeout
> parameters (e.g. parameter1 >= parameter2) ? The documentation of each
> package seems to only explain its own configuration directives, but I'm
> thinking our problems may be related to some not-so-good combination of
> timeout settings (at the moment they all have values > 0 and < infinite
> though).
>
> Thx,
>
> Pieter
>
>
> > noodl
> >
> > ---------------------------------------------------------------------
> > The official User-To-User support forum of the Apache HTTP Server
Project.
> > See <URL:http://httpd.apache.org/userslist.html> for more info.
> > To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> > " from the digest: users-digest-unsubscribe@httpd.apache.org
> > For additional commands, e-mail: users-help@httpd.apache.org
> >
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
RE: [users@httpd] Server hanging with requests in "W" state
Posted by Narendra Verma <na...@impetus.co.in>.
Thanks for replying to me.
But I am not much aware about apache.
I started it just before sometime ago.
So please can you confirm it how can I do following:-
Apache --- mod_jk connector ---tomcat
Caching by mod_disk_cache and mod_cache
Now if I changed any static file at tomcat(that is already cached at apache)
must be re-cached in further request without shutting down apache and
tomcat.
Is it possible ? if so please can you give me guide me for configuration the
same?
Thanks a lot..
-----Original Message-----
From: David Cassidy [mailto:david@twocats.co.uk]
Sent: Tuesday, February 26, 2008 7:20 PM
To: users@httpd.apache.org
Subject: Re: [users@httpd] Server hanging with requests in "W" state
Whats your timeout and keepalivetimeout settings set to ?
If you have them more than 15 secs ? then you'll DOS yourself
IE will hold open connections to the server for as long as it can...
On Tue, 2008-02-26 at 09:29 +0100, pthyseba@intec.ugent.be wrote:
> Hello,
>
> > On 25/02/2008, pthyseba@intec.ugent.be <pt...@intec.ugent.be> wrote:
> >> We're having some strange problems with our webservers (https), which
> >> are
> >> Apache 2.0.52 on RHEL4 and 2.0.46 on RHEL 3.
> >
> > This could be due to MSIE's duff SSL implementation. Do you have
> > something like this in your SSL config?
> >
> > BrowserMatch ".*MSIE.*" \
> > nokeepalive ssl-unclean-shutdown \
> > downgrade-1.0 force-response-1.0
> >
> >
>
>
> We do have this kind of line in our SSL configuration. However, we have
> also seen this problem (512 W's with increasingly high "SS" values in
> server-status, until even server-status can no longer be requested) on our
> non-ssl web servers, albeit a lot less frequently (possibly because of
> lower load triggering our scenario less frequently).
>
> I'm wondering what the technical (apache, sockets, OS,...) reason is for
> the absence of some "Apache kill switch" which would terminate (that is,
> free the apache slot and make it available to another client) a connection
> with an "SS" value greater than some threshold - or is this supposed to be
> solved by any of the apache Timeout settings?
>
> If not, I'll have to dig further into the plugins we're running.
>
> Is there a good overview available which explains the relationship between
> Apache timeout settings and e.g. worker.properties (mod_jk) timeout
> parameters (e.g. parameter1 >= parameter2) ? The documentation of each
> package seems to only explain its own configuration directives, but I'm
> thinking our problems may be related to some not-so-good combination of
> timeout settings (at the moment they all have values > 0 and < infinite
> though).
>
> Thx,
>
> Pieter
>
>
> > noodl
> >
> > ---------------------------------------------------------------------
> > The official User-To-User support forum of the Apache HTTP Server
Project.
> > See <URL:http://httpd.apache.org/userslist.html> for more info.
> > To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> > " from the digest: users-digest-unsubscribe@httpd.apache.org
> > For additional commands, e-mail: users-help@httpd.apache.org
> >
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Server hanging with requests in "W" state
Posted by David Cassidy <da...@twocats.co.uk>.
Whats your timeout and keepalivetimeout settings set to ?
If you have them more than 15 secs ? then you'll DOS yourself
IE will hold open connections to the server for as long as it can...
On Tue, 2008-02-26 at 09:29 +0100, pthyseba@intec.ugent.be wrote:
> Hello,
>
> > On 25/02/2008, pthyseba@intec.ugent.be <pt...@intec.ugent.be> wrote:
> >> We're having some strange problems with our webservers (https), which
> >> are
> >> Apache 2.0.52 on RHEL4 and 2.0.46 on RHEL 3.
> >
> > This could be due to MSIE's duff SSL implementation. Do you have
> > something like this in your SSL config?
> >
> > BrowserMatch ".*MSIE.*" \
> > nokeepalive ssl-unclean-shutdown \
> > downgrade-1.0 force-response-1.0
> >
> >
>
>
> We do have this kind of line in our SSL configuration. However, we have
> also seen this problem (512 W's with increasingly high "SS" values in
> server-status, until even server-status can no longer be requested) on our
> non-ssl web servers, albeit a lot less frequently (possibly because of
> lower load triggering our scenario less frequently).
>
> I'm wondering what the technical (apache, sockets, OS,...) reason is for
> the absence of some "Apache kill switch" which would terminate (that is,
> free the apache slot and make it available to another client) a connection
> with an "SS" value greater than some threshold - or is this supposed to be
> solved by any of the apache Timeout settings?
>
> If not, I'll have to dig further into the plugins we're running.
>
> Is there a good overview available which explains the relationship between
> Apache timeout settings and e.g. worker.properties (mod_jk) timeout
> parameters (e.g. parameter1 >= parameter2) ? The documentation of each
> package seems to only explain its own configuration directives, but I'm
> thinking our problems may be related to some not-so-good combination of
> timeout settings (at the moment they all have values > 0 and < infinite
> though).
>
> Thx,
>
> Pieter
>
>
> > noodl
> >
> > ---------------------------------------------------------------------
> > The official User-To-User support forum of the Apache HTTP Server Project.
> > See <URL:http://httpd.apache.org/userslist.html> for more info.
> > To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> > " from the digest: users-digest-unsubscribe@httpd.apache.org
> > For additional commands, e-mail: users-help@httpd.apache.org
> >
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Server hanging with requests in "W" state
Posted by pt...@intec.ugent.be.
Hello,
> On 25/02/2008, pthyseba@intec.ugent.be <pt...@intec.ugent.be> wrote:
>> We're having some strange problems with our webservers (https), which
>> are
>> Apache 2.0.52 on RHEL4 and 2.0.46 on RHEL 3.
>
> This could be due to MSIE's duff SSL implementation. Do you have
> something like this in your SSL config?
>
> BrowserMatch ".*MSIE.*" \
> nokeepalive ssl-unclean-shutdown \
> downgrade-1.0 force-response-1.0
>
>
We do have this kind of line in our SSL configuration. However, we have
also seen this problem (512 W's with increasingly high "SS" values in
server-status, until even server-status can no longer be requested) on our
non-ssl web servers, albeit a lot less frequently (possibly because of
lower load triggering our scenario less frequently).
I'm wondering what the technical (apache, sockets, OS,...) reason is for
the absence of some "Apache kill switch" which would terminate (that is,
free the apache slot and make it available to another client) a connection
with an "SS" value greater than some threshold - or is this supposed to be
solved by any of the apache Timeout settings?
If not, I'll have to dig further into the plugins we're running.
Is there a good overview available which explains the relationship between
Apache timeout settings and e.g. worker.properties (mod_jk) timeout
parameters (e.g. parameter1 >= parameter2) ? The documentation of each
package seems to only explain its own configuration directives, but I'm
thinking our problems may be related to some not-so-good combination of
timeout settings (at the moment they all have values > 0 and < infinite
though).
Thx,
Pieter
> noodl
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Server hanging with requests in "W" state
Posted by Vincent Bray <no...@gmail.com>.
On 25/02/2008, pthyseba@intec.ugent.be <pt...@intec.ugent.be> wrote:
> We're having some strange problems with our webservers (https), which are
> Apache 2.0.52 on RHEL4 and 2.0.46 on RHEL 3.
This could be due to MSIE's duff SSL implementation. Do you have
something like this in your SSL config?
BrowserMatch ".*MSIE.*" \
nokeepalive ssl-unclean-shutdown \
downgrade-1.0 force-response-1.0
--
noodl
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org