You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@httpd.apache.org by Yannis Tsakiris <gt...@dolnet.gr> on 2005/03/28 14:27:24 UTC
[users@httpd] Strange Problem
Hi there,
This is a problem I'm facing for a long time.
I have a server which serves advertisement banners (with phpAdsNew). The apache web server is the only application this server runs (there is no database server or other service running).
The machine consists of two pIII CPUs at 1000MHz and is equiped with 1 GB of RAM.
It serves approximately 300,000 requests per hour at rush hour.
The load average reaches a peak of 0.50 at this load.
So far so good, but in an arbitrary moment all apache processes stop serving requests, and all of the processes are trying to consume a lot of cpu time, and load average is increasing very fast (at the end the load average is almost equal to the number of running apache processes, which is 500). In this condition the server is not responding and all I can do is to reboot the machine.
When this strange thing happens varies. It can happen after a few hours of uptime, or it can suddenly happen after 20 days since the last reboot. Anyway, it is ok to say that this thing happens about once in a week.
I have read the logs, both apache's and system's but there is nothing. Not a clue. Everything seems to stop at the time of the "crash". All logs stop at this moment without a error message.
There is a crontab that restarts the apache server every day, late at night when the traffic is in its lowest. (but this does not seem to help...)
The versions of the software I'm using is: SuSE Linux 9.1 , Linux kernel 2.4.21-99-smp4G (the precompiled kernel that comes with SuSE 9.1), Apache 2.0.52 , PHP 4.3.11, Turck mm-cache 2.4.6
Any ideas? Your help will be very much appreciated.
Yannis Tsakiris
Re: [users@httpd] Strange Problem
Posted by ma...@morningstarcom.net.
On Monday 28 March 2005 07:27, Yannis Tsakiris wrote:
> In this condition the server is not responding and all I can do is to
> reboot the machine.
It sounds like you have a problem with your CPU getting hot and the box just
stops.
I have two boxes with the same mobo and both of them did what you describe.
Adding more air flow to help keep the box cool stopped the problem. This may
or may not be your problem.
--
Win a Vespa Scooter, $3,000.00 value
or a Dell Gift Card, $2,000.00 value
http://www.morningstarcom.net/raffle-contest.php
--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.
MailScanner thanks transtec Computers for their support.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: Re[2]: [users@httpd] Strange Problem
Posted by Joshua Slive <js...@gmail.com>.
On Mon, 28 Mar 2005 20:05:53 +0300, John <is...@cc.uoi.gr> wrote:
> Couldn't it be a configuration limit in the httpd.conf ? (Such as
> maxconnection, etc)
Not likely. His symptoms indicate spinning processes using lots of CPU.
Joshua.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re[2]: [users@httpd] Strange Problem
Posted by John <is...@cc.uoi.gr>.
From: Joshua Slive <js...@gmail.com>
To: users@httpd.apache.org
Date: Monday, March 28, 2005, 6:44:08 PM
Subject: [users@httpd] Strange Problem
Monday, March 28, 2005, 6:44:08 PM, you wrote:
> On Mon, 28 Mar 2005 15:27:24 +0300, Yannis Tsakiris <gt...@dolnet.gr> wrote:
>> So far so good, but in an arbitrary moment all apache processes stop serving
>> requests, and all of the processes are trying to consume a lot of cpu time,
>> and load average is increasing very fast (at the end the load average is
>> almost equal to the number of running apache processes, which is 500). In
>> this condition the server is not responding and all I can do is to reboot
>> the machine.
> If you can use the shell, then you should do some debugging: attach to
> the processes with gdb and strace and see what they are up to. It is
> unlikely that this is an apache-httpd problem. More likely one of
> your scripts is getting stuck someplace.
> Joshua.
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
Couldn't it be a configuration limit in the httpd.conf ? (Such as
maxconnection, etc)
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: Re[2]: [users@httpd] Strange Problem
Posted by Joshua Slive <js...@gmail.com>.
On 4/14/05, John <is...@cc.uoi.gr> wrote:
> How do you use gdb with apache process and strace ?
See, for example,
http://httpd.apache.org/dev/debugging.html
Joshua.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re[2]: [users@httpd] Strange Problem
Posted by John <is...@cc.uoi.gr>.
How do you use gdb with apache process and strace ?
From: herauthon <he...@home.nl>
To: <us...@httpd.apache.org>, "Joshua Slive" <js...@gmail.com>
Date: Thursday, April 14, 2005, 1:18:56 AM
Subject: [users@httpd] Strange Problem
Thursday, April 14, 2005, 1:18:56 AM, you wrote:
> <checking>
> is this issue solved
> and does apache use garbage collect mechanisms?
> ----- Original Message -----
> From: "Joshua Slive" <js...@gmail.com>
> To: <us...@httpd.apache.org>
> Sent: Monday, March 28, 2005 5:44 PM
> Subject: Re: [users@httpd] Strange Problem
>> On Mon, 28 Mar 2005 15:27:24 +0300, Yannis Tsakiris <gt...@dolnet.gr>
> wrote:
>> > So far so good, but in an arbitrary moment all apache processes stop
> serving
>> > requests, and all of the processes are trying to consume a lot of cpu
> time,
>> > and load average is increasing very fast (at the end the load average is
>> > almost equal to the number of running apache processes, which is 500).
> In
>> > this condition the server is not responding and all I can do is to
> reboot
>> > the machine.
>>
>> If you can use the shell, then you should do some debugging: attach to
>> the processes with gdb and strace and see what they are up to. It is
>> unlikely that this is an apache-httpd problem. More likely one of
>> your scripts is getting stuck someplace.
>>
>> Joshua.
>>
>> ---------------------------------------------------------------------
>> The official User-To-User support forum of the Apache HTTP Server Project.
>> See <URL:http://httpd.apache.org/userslist.html> for more info.
>> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
>> " from the digest: users-digest-unsubscribe@httpd.apache.org
>> For additional commands, e-mail: users-help@httpd.apache.org
>>
>>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Strange Problem
Posted by herauthon <he...@home.nl>.
<checking>
is this issue solved
and does apache use garbage collect mechanisms?
----- Original Message -----
From: "Joshua Slive" <js...@gmail.com>
To: <us...@httpd.apache.org>
Sent: Monday, March 28, 2005 5:44 PM
Subject: Re: [users@httpd] Strange Problem
> On Mon, 28 Mar 2005 15:27:24 +0300, Yannis Tsakiris <gt...@dolnet.gr>
wrote:
> > So far so good, but in an arbitrary moment all apache processes stop
serving
> > requests, and all of the processes are trying to consume a lot of cpu
time,
> > and load average is increasing very fast (at the end the load average is
> > almost equal to the number of running apache processes, which is 500).
In
> > this condition the server is not responding and all I can do is to
reboot
> > the machine.
>
> If you can use the shell, then you should do some debugging: attach to
> the processes with gdb and strace and see what they are up to. It is
> unlikely that this is an apache-httpd problem. More likely one of
> your scripts is getting stuck someplace.
>
> Joshua.
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Strange Problem
Posted by Joshua Slive <js...@gmail.com>.
On Mon, 28 Mar 2005 15:27:24 +0300, Yannis Tsakiris <gt...@dolnet.gr> wrote:
> So far so good, but in an arbitrary moment all apache processes stop serving
> requests, and all of the processes are trying to consume a lot of cpu time,
> and load average is increasing very fast (at the end the load average is
> almost equal to the number of running apache processes, which is 500). In
> this condition the server is not responding and all I can do is to reboot
> the machine.
If you can use the shell, then you should do some debugging: attach to
the processes with gdb and strace and see what they are up to. It is
unlikely that this is an apache-httpd problem. More likely one of
your scripts is getting stuck someplace.
Joshua.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Strange Problem
Posted by Yannis Tsakiris <gt...@dolnet.gr>.
> > This is a problem I'm facing for a long time.
> > So far so good, but in an arbitrary moment all apache processes
> > stop serving requests, and all of the processes are trying to
> > consume a lot of cpu time, and load average is increasing very fast
> > ...
> How many servers are busy during the crash? (if you are monitoring
> Apache status through mrtg graphs)
> Anything in /var/log/messsages, warn ?
> It can be a hardware problem that everything hangs.
During the crash and using ps or top I can see that all apache processes are
busy (running).
In the logs there is nothing strange, just normal messages which stop at the
time of the "crash" (even though this is not really a crash, since apache is
still running).
There are two identical servers (exactly the same hardware, exactly the same
software installed) and the problem occurs in both server (I don't know
*when* it will happen, and in *which* computer, it can happen at anytime in
any of the two computes), and that makes me think that it's highly unlikely
to be a hardware failure (it would be a great coincidence if two identical
machines have the same hardware problem).
Since all apache processes are busy makes be believe that they all fall in a
"sleep-less" infinite loop. The question is why they do so, and which of the
software is erroneous...
Any ideas..?
Yannis Tsakiris
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Strange Problem
Posted by John <is...@cc.uoi.gr>.
From: Yannis Tsakiris <gt...@dolnet.gr>
To: <us...@httpd.apache.org>
Date: Monday, March 28, 2005, 3:27:24 PM
Subject: [users@httpd] Strange Problem
Monday, March 28, 2005, 3:27:24 PM, you wrote:
> Hi there,
> This is a problem I'm facing for a long time.
> I have a server which serves advertisement banners (with
> phpAdsNew). The apache web server is the only application this
> server runs (there is no database server or other service running).
> The machine consists of two pIII CPUs at 1000MHz and is equiped with 1 GB of RAM.
> It serves approximately 300,000 requests per hour at rush hour.
> The load average reaches a peak of 0.50 at this load.
> So far so good, but in an arbitrary moment all apache processes
> stop serving requests, and all of the processes are trying to
> consume a lot of cpu time, and load average is increasing very fast
> (at the end the load average is almost equal to the number of
> running apache processes, which is 500). In this condition the
> server is not responding and all I can do is to reboot the machine.
> When this strange thing happens varies. It can happen after a
> few hours of uptime, or it can suddenly happen after 20 days since
> the last reboot. Anyway, it is ok to say that this thing happens
> about once in a week.
> I have read the logs, both apache's and system's but there is
> nothing. Not a clue. Everything seems to stop at the time of the
> "crash". All logs stop at this moment without a error message.
> There is a crontab that restarts the apache server every day,
> late at night when the traffic is in its lowest. (but this does not
> seem to help...)
> The versions of the software I'm using is: SuSE Linux 9.1 ,
> Linux kernel 2.4.21-99-smp4G (the precompiled kernel that comes with
> SuSE 9.1), Apache 2.0.52 , PHP 4.3.11, Turck mm-cache 2.4.6
> Any ideas? Your help will be very much appreciated.
> Yannis Tsakiris
>
How many servers are busy during the crash? (if you are monitoring
Apache status through mrtg graphs)
Anything in /var/log/messsages, warn ?
It can be a hardware problem that everything hangs.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org