You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by Alec Matusis <ma...@yahoo.com> on 2007/10/01 08:32:05 UTC

2.0.54 unstable, requests time-out, NO warnings in logs

We are running a busy Apache/2.0.54 server on 2.6.9 kernel, that suddenly becomes very slow- requests either time out, or it takes 10-20sec to serve a 1K thumbnail. 
It is somewhat correlated with load spikes, but not perfectly (by looking at the bandwidth graph, it never happens during the low bandwidth periods at night, but it does not coincide with peaks of b/w) 

When we initially encountered an apache overload, it was always accompanied with 

[error] server reached MaxClients setting, consider raising the MaxClients setting 

in the apache error log. We also got 

kernel: possible SYN flooding on port 80. Sending cookies. 

in /var/log/messages system log. 

After that I raised MaxClients from 200 to 300. The problem initially disappeared, but after our bandwidth grew a bit more, we got this behavior again. 
Now apache crashes (becomes very slow) silently, with no warning in apache error logs at all (although we still get SYN flood message in the system log) 
When apache is this 'slow' regime, /server-status still shows available slots, i.e. MaxClients is not reached. 

This is the relevant part of httpd.conf: 

ServerLimit 300 
# we are using prefork MPM 
StartServers 10 
MinSpareServers 5 
MaxSpareServers 20 
MaxClients 300 
MaxRequestsPerChild 10000 
MaxMemFree 2500 

The server has 4GB of physical RAM and 4GB of swap. During these apache “slowdowns", the swap size is still 0 and vmstat shows no swapping at all. 
I suspect the problem may be in 

MaxMemFree 2500 

but then I would expect some kind of  “out of memory” errors in the logs? 

I am posting it on this list since I have not gotten a response in the users list, and I think it's a bug for two reasons: 

1) When apache is in this slow "degraded" regime, I would expect a log message in the apache error log, with an explanation why. 

3) If this is related to resource exhaustion, I would expect the server to recover from this regime by itself when the load subsides, but this is not the case. Only apachectl start/stop recovers the server.

RE: 2.0.54 unstable, requests time-out, NO warnings in logs

Posted by Alec Matusis <ma...@yahoo.com>.
> Looks like an overloaded server to me. It is strange that there are so
> many processes in 'R' state. This looks to me as some clients are
> playing
> games with your server by connecting it but not sending a request.

The thing is, as soon as apache is stop/started, the R's disappear- if it
was an attack, I would not expect it to stop as soon as we restart apache. 

> -----Original Message-----
> From: Ruediger Pluem [mailto:rpluem@apache.org]
> Sent: Tuesday, October 02, 2007 1:46 AM
> To: dev@httpd.apache.org
> Subject: Re: 2.0.54 unstable, requests time-out, NO warnings in logs
> 
> 
> 
> On 10/02/2007 01:04 AM, Alec Matusis wrote:
> >> Have you checked without the MaxMemFree setting?
> >> Why do you use MaxMemFree with such a small value at all?
> >
> > I finally removed MaxMemFree altogether, and it crashed again.
> Nothing in
> > the apache error logs, but this is how /server-status looks like
> during the
> > crash:
> >
> > 300 requests currently being processed, 0 idle workers
> > WCRRRRRRCRRRRRRRRCCRRRRRRRRRRRRRRRCCRWWRRRCRWRRCRRCRWRRWRRRRRRRC
> > RRCRRRRCRCRRWRRRRWWCRRRWRRRWRCRRRCRRRRRRRRRCRRCRRRCCCRRCRRRRCCWR
> > RCRRRRRRRRRRRRRRRRWWRRRWRWRWCCCWWWRCRRRRRCRCCWRRWRCRCRWRRRRCRRRR
> > RRRRWRRRWRRRCRRCRRCRRRCRRRRRRRRRRRRRRRRCCCRWRRRRRRRRRRRRCRRRCRRR
> > RRCRRRRRRRRCRWCRCRRRRRRRWWCWRRCWRRRCRRCRRRCR
> 
> Looks like an overloaded server to me. It is strange that there are so
> many processes in 'R' state. This looks to me as some clients are
> playing
> games with your server by connecting it but not sending a request.
> 
> 
> >
> > Immediately after I restarted the apache after the crash, I did get
> >
> > [Mon Oct 01 15:20:49 2007] [notice] mod_python: Creating 32 session
> mutexes
> > based on 300 max processes and 0 max threads.
> > [Mon Oct 01 15:20:49 2007] [notice] Apache/2.0.54 (Unix)
> mod_python/3.1.4
> > Python/2.4.1 configured -- resuming normal operations
> > ****[Mon Oct 01 15:21:25 2007] [error] server reached MaxClients
> setting,
> > consider raising the MaxClients setting***
> >
> > but it's strange that this message was not written before or during
> the
> > crash, even though /server-status shows no available free child
> processes.
> 
> It is only written once during the livetime of the server. So in your
> case
> above: As it appeared immediately after the start of the server it will
> never appear again until you restart the server.
> 
> Regards
> 
> Rüdiger

Re: 2.0.54 unstable, requests time-out, NO warnings in logs

Posted by Ruediger Pluem <rp...@apache.org>.

On 10/02/2007 01:04 AM, Alec Matusis wrote:
>> Have you checked without the MaxMemFree setting?
>> Why do you use MaxMemFree with such a small value at all?
> 
> I finally removed MaxMemFree altogether, and it crashed again. Nothing in
> the apache error logs, but this is how /server-status looks like during the
> crash:
> 
> 300 requests currently being processed, 0 idle workers 
> WCRRRRRRCRRRRRRRRCCRRRRRRRRRRRRRRRCCRWWRRRCRWRRCRRCRWRRWRRRRRRRC
> RRCRRRRCRCRRWRRRRWWCRRRWRRRWRCRRRCRRRRRRRRRCRRCRRRCCCRRCRRRRCCWR
> RCRRRRRRRRRRRRRRRRWWRRRWRWRWCCCWWWRCRRRRRCRCCWRRWRCRCRWRRRRCRRRR
> RRRRWRRRWRRRCRRCRRCRRRCRRRRRRRRRRRRRRRRCCCRWRRRRRRRRRRRRCRRRCRRR
> RRCRRRRRRRRCRWCRCRRRRRRRWWCWRRCWRRRCRRCRRRCR

Looks like an overloaded server to me. It is strange that there are so
many processes in 'R' state. This looks to me as some clients are playing
games with your server by connecting it but not sending a request.


> 
> Immediately after I restarted the apache after the crash, I did get
> 
> [Mon Oct 01 15:20:49 2007] [notice] mod_python: Creating 32 session mutexes
> based on 300 max processes and 0 max threads.
> [Mon Oct 01 15:20:49 2007] [notice] Apache/2.0.54 (Unix) mod_python/3.1.4
> Python/2.4.1 configured -- resuming normal operations
> ****[Mon Oct 01 15:21:25 2007] [error] server reached MaxClients setting,
> consider raising the MaxClients setting***
> 
> but it's strange that this message was not written before or during the
> crash, even though /server-status shows no available free child processes.

It is only written once during the livetime of the server. So in your case
above: As it appeared immediately after the start of the server it will
never appear again until you restart the server.

Regards

Rüdiger

RE: 2.0.54 unstable, requests time-out, NO warnings in logs

Posted by Alec Matusis <ma...@yahoo.com>.
> Have you checked without the MaxMemFree setting?
> Why do you use MaxMemFree with such a small value at all?

I finally removed MaxMemFree altogether, and it crashed again. Nothing in
the apache error logs, but this is how /server-status looks like during the
crash:

300 requests currently being processed, 0 idle workers 
WCRRRRRRCRRRRRRRRCCRRRRRRRRRRRRRRRCCRWWRRRCRWRRCRRCRWRRWRRRRRRRC
RRCRRRRCRCRRWRRRRWWCRRRWRRRWRCRRRCRRRRRRRRRCRRCRRRCCCRRCRRRRCCWR
RCRRRRRRRRRRRRRRRRWWRRRWRWRWCCCWWWRCRRRRRCRCCWRRWRCRCRWRRRRCRRRR
RRRRWRRRWRRRCRRCRRCRRRCRRRRRRRRRRRRRRRRCCCRWRRRRRRRRRRRRCRRRCRRR
RRCRRRRRRRRCRWCRCRRRRRRRWWCWRRCWRRRCRRCRRRCR

Immediately after I restarted the apache after the crash, I did get

[Mon Oct 01 15:20:49 2007] [notice] mod_python: Creating 32 session mutexes
based on 300 max processes and 0 max threads.
[Mon Oct 01 15:20:49 2007] [notice] Apache/2.0.54 (Unix) mod_python/3.1.4
Python/2.4.1 configured -- resuming normal operations
****[Mon Oct 01 15:21:25 2007] [error] server reached MaxClients setting,
consider raising the MaxClients setting***

but it's strange that this message was not written before or during the
crash, even though /server-status shows no available free child processes.


> -----Original Message-----
> From: Ruediger Pluem [mailto:rpluem@apache.org]
> Sent: Monday, October 01, 2007 1:23 AM
> To: dev@httpd.apache.org
> Subject: Re: 2.0.54 unstable, requests time-out, NO warnings in logs
> 
> 
> 
> On 10/01/2007 08:32 AM, Alec Matusis wrote:
> > We are running a busy Apache/2.0.54 server on 2.6.9 kernel, that
> suddenly becomes very slow- requests either time out, or it takes 10-
> 20sec to serve a 1K thumbnail.
> > It is somewhat correlated with load spikes, but not perfectly (by
> looking at the bandwidth graph, it never happens during the low
> bandwidth periods at night, but it does not coincide with peaks of b/w)
> >
> > When we initially encountered an apache overload, it was always
> accompanied with
> >
> > [error] server reached MaxClients setting, consider raising the
> MaxClients setting
> >
> > in the apache error log. We also got
> >
> > kernel: possible SYN flooding on port 80. Sending cookies.
> >
> > in /var/log/messages system log.
> >
> > After that I raised MaxClients from 200 to 300. The problem initially
> disappeared, but after our bandwidth grew a bit more, we got this
> behavior again.
> > Now apache crashes (becomes very slow) silently, with no warning in
> apache error logs at all (although we still get SYN flood message in
> the system log)
> > When apache is this 'slow' regime, /server-status still shows
> available slots, i.e. MaxClients is not reached.
> >
> > This is the relevant part of httpd.conf:
> >
> > ServerLimit 300
> > # we are using prefork MPM
> > StartServers 10
> > MinSpareServers 5
> > MaxSpareServers 20
> > MaxClients 300
> > MaxRequestsPerChild 10000
> > MaxMemFree 2500
> >
> > The server has 4GB of physical RAM and 4GB of swap. During these
> apache “slowdowns", the swap size is still 0 and vmstat shows no
> swapping at all.
> > I suspect the problem may be in
> >
> > MaxMemFree 2500
> 
> Have you checked without the MaxMemFree setting?
> Why do you use MaxMemFree with such a small value at all?
> 
> Regards
> 
> Rüdiger

RE: 2.0.54 unstable, requests time-out, NO warnings in logs

Posted by Alec Matusis <ma...@yahoo.com>.
> Have you checked without the MaxMemFree setting?
I raised MaxMemFree to 3100, we will have to wait for a few days, since it
does not happen every day.

> Why do you use MaxMemFree with such a small value at all?

We are also running 2 twistd server processes on this machine that can take
up to 256MB each, so we wanted to leave some room for them.

As I understand from the manual, MaxMemFree is not an absolute limit on the
memory available to Apache anyway, it just asks it to release *unused*
memory with free() call after it reaches MaxMemFree value? 

> -----Original Message-----
> From: Ruediger Pluem [mailto:rpluem@apache.org]
> Sent: Monday, October 01, 2007 1:23 AM
> To: dev@httpd.apache.org
> Subject: Re: 2.0.54 unstable, requests time-out, NO warnings in logs
> 
> 
> 
> On 10/01/2007 08:32 AM, Alec Matusis wrote:
> > We are running a busy Apache/2.0.54 server on 2.6.9 kernel, that
> suddenly becomes very slow- requests either time out, or it takes 10-
> 20sec to serve a 1K thumbnail.
> > It is somewhat correlated with load spikes, but not perfectly (by
> looking at the bandwidth graph, it never happens during the low
> bandwidth periods at night, but it does not coincide with peaks of b/w)
> >
> > When we initially encountered an apache overload, it was always
> accompanied with
> >
> > [error] server reached MaxClients setting, consider raising the
> MaxClients setting
> >
> > in the apache error log. We also got
> >
> > kernel: possible SYN flooding on port 80. Sending cookies.
> >
> > in /var/log/messages system log.
> >
> > After that I raised MaxClients from 200 to 300. The problem initially
> disappeared, but after our bandwidth grew a bit more, we got this
> behavior again.
> > Now apache crashes (becomes very slow) silently, with no warning in
> apache error logs at all (although we still get SYN flood message in
> the system log)
> > When apache is this 'slow' regime, /server-status still shows
> available slots, i.e. MaxClients is not reached.
> >
> > This is the relevant part of httpd.conf:
> >
> > ServerLimit 300
> > # we are using prefork MPM
> > StartServers 10
> > MinSpareServers 5
> > MaxSpareServers 20
> > MaxClients 300
> > MaxRequestsPerChild 10000
> > MaxMemFree 2500
> >
> > The server has 4GB of physical RAM and 4GB of swap. During these
> apache “slowdowns", the swap size is still 0 and vmstat shows no
> swapping at all.
> > I suspect the problem may be in
> >
> > MaxMemFree 2500
> 
> Have you checked without the MaxMemFree setting?
> Why do you use MaxMemFree with such a small value at all?
> 
> Regards
> 
> Rüdiger

Re: 2.0.54 unstable, requests time-out, NO warnings in logs

Posted by Ruediger Pluem <rp...@apache.org>.

On 10/01/2007 08:32 AM, Alec Matusis wrote:
> We are running a busy Apache/2.0.54 server on 2.6.9 kernel, that suddenly becomes very slow- requests either time out, or it takes 10-20sec to serve a 1K thumbnail. 
> It is somewhat correlated with load spikes, but not perfectly (by looking at the bandwidth graph, it never happens during the low bandwidth periods at night, but it does not coincide with peaks of b/w) 
> 
> When we initially encountered an apache overload, it was always accompanied with 
> 
> [error] server reached MaxClients setting, consider raising the MaxClients setting 
> 
> in the apache error log. We also got 
> 
> kernel: possible SYN flooding on port 80. Sending cookies. 
> 
> in /var/log/messages system log. 
> 
> After that I raised MaxClients from 200 to 300. The problem initially disappeared, but after our bandwidth grew a bit more, we got this behavior again. 
> Now apache crashes (becomes very slow) silently, with no warning in apache error logs at all (although we still get SYN flood message in the system log) 
> When apache is this 'slow' regime, /server-status still shows available slots, i.e. MaxClients is not reached. 
> 
> This is the relevant part of httpd.conf: 
> 
> ServerLimit 300 
> # we are using prefork MPM 
> StartServers 10 
> MinSpareServers 5 
> MaxSpareServers 20 
> MaxClients 300 
> MaxRequestsPerChild 10000 
> MaxMemFree 2500 
> 
> The server has 4GB of physical RAM and 4GB of swap. During these apache “slowdowns", the swap size is still 0 and vmstat shows no swapping at all. 
> I suspect the problem may be in 
> 
> MaxMemFree 2500 

Have you checked without the MaxMemFree setting?
Why do you use MaxMemFree with such a small value at all?

Regards

Rüdiger