You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@httpd.apache.org by Stut <st...@gmail.com> on 2008/06/13 23:30:58 UTC
[users@httpd] High load, low CPU and processes in state "devfs"
Hi,
I have a problem with one of the web servers I manage. It runs FreeBSD
6.2, Apache 2.2.8 and PHP 5.2.1. It runs a high-ish number of pre-fork
processes (usually around 240).
What basically happens is that during our peak hours in the evening
the site becomes very slow as does everything on the server. The top
output shows nearly all httpd processes in the state "devfs" and the
load is jumping up over 10 but the CPU is 70% idle. There's plenty of
free memory, over 4GB, and lots of available disk space.
While it's in this state all disk access is painfully slow. I've
checked all the drives and the RAID card status and everything appears
to be fine. It also recovers itself after a few hours when the traffic
dies down again.
I've spent a lot of today Googling but can't find any reference to
this particular combination of symptoms. Does anyone have any ideas?
Thanks.
-Stut
--
http://stut.net/
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] High load, low CPU and processes in state "devfs"
Posted by Mohit Anchlia <mo...@gmail.com>.
You'll need to analyze this data with what you are seeing on this system. I
suggest you get sar like tool. Look at http://www.googlebit.com/bsdsar/ or
see this post too
http://groups.google.com/group/sol.lists.freebsd.questions/browse_thread/thread/b999f095612e59bd/257ddbadf0c9ec71?hl=en&lnk=st&q=FreeBSD+sar#257ddbadf0c9ec71
Or ask you system admin. You need to look at it very closely.
On Sat, Jun 14, 2008 at 10:23 AM, Stut <st...@gmail.com> wrote:
> Thanks Mohit. Shortly after you sent this last night I enabled system
> accounting. There's now a fair amount of data in the output but I'm not
> really sure where to find the disk wait times...
>
> root@harold:~# sa
> 41065 44351.99re 428.71cp 11avio 823k
> 8678 33375.24re 418.23cp 34avio 496k httpd*
> 2469 51.19re 4.79cp 2avio 767k convert
> 1513 1180.51re 2.69cp 48avio 4929k php
> 2469 56.01re 2.59cp 2avio 1312k composite
> 590 3945.74re 0.09cp 5avio 6182k smtp
> 624 152.45re 0.09cp 0avio 196079k perl*
> 46 468.10re 0.08cp 691avio 574k cleanup
> 1170 10.05re 0.08cp 11avio 107305k rateup
> 47 388.52re 0.03cp 190avio 1054k pickup
> 743 240.80re 0.02cp 0avio 27976k
> trivial-rewrite
> 10443 1319.75re 0.00cp 0avio 26333351k sh
> 154 305.90re 0.00cp 24avio 44750k bounce
> 4454 101.21re 0.00cp 0avio 1753042k sendmail
> 4454 52.77re 0.00cp 7avio 2748826k postdrop
> 2 1.00re 0.00cp 0avio 3136k top
> 9 1278.02re 0.00cp 2avio 32944k ***other
> 389 13.73re 0.00cp 0avio 908587k ps
> 2 0.01re 0.00cp 6avio 981k sa
> 24 80.90re 0.00cp 0avio 60224k scache
> 9 35.12re 0.00cp 0avio 12700k anvil
> 78 0.28re 0.00cp 0avio 198900k atrun
> 623 1043.23re 0.00cp 0avio 959100k cron*
> 39 1.24re 0.00cp 2avio 112100k dd
> 27 52.78re 0.00cp 0avio 109600k error
> 65 114.24re 0.00cp 0avio 298000k flush
> 1165 41.51re 0.00cp 0avio 4475200k grep
> 40 0.14re 0.00cp 0avio 60000k jot
> 279 1.16re 0.00cp 3avio 971600k mv
> 7 0.31re 0.00cp 3avio 7200k newsyslog
> 13 1.68re 0.00cp 0avio 42500k proxymap
> 12 23.78re 0.00cp 0avio 78800k smtpd
> 40 0.16re 0.00cp 2avio 194800k unlink
> 388 14.47re 0.00cp 0avio 1219700k wc
>
> Any help would be appreciated. Thanks.
>
> -Stut
>
>
> On 14 Jun 2008, at 00:43, Mohit Anchlia wrote:
>
> try "sa"
>>
>> On 6/13/08, Stut <st...@gmail.com> wrote: On 13 Jun 2008, at 23:38,
>> Mohit Anchlia wrote:
>> look at "sar" output and see the wait times on the disk is more than the
>> service time.
>>
>> That command doesn't seem to exist on FreeBSD. Do you know if there's an
>> equivalent command?
>>
>> -Stut
>>
>>
>> On 6/13/08, Stut <st...@gmail.com> wrote:
>>
>> On 13 Jun 2008, at 22:56, Dragon wrote:
>>
>> Stut wrote:
>> On 13 Jun 2008, at 22:37, Dragon wrote:
>> Stut wrote:
>> Hi,
>>
>> I have a problem with one of the web servers I manage. It runs
>> FreeBSD
>> 6.2, Apache 2.2.8 and PHP 5.2.1. It runs a high-ish number of pre- fork
>> processes (usually around 240).
>>
>> What basically happens is that during our peak hours in the evening
>> the site becomes very slow as does everything on the server. The top
>> output shows nearly all httpd processes in the state "devfs" and the
>> load is jumping up over 10 but the CPU is 70% idle. There's plenty of
>> free memory, over 4GB, and lots of available disk space.
>>
>> While it's in this state all disk access is painfully slow. I've
>> checked all the drives and the RAID card status and everything
>> appears
>> to be fine. It also recovers itself after a few hours when the
>> traffic
>> dies down again.
>>
>> I've spent a lot of today Googling but can't find any reference to
>> this particular combination of symptoms. Does anyone have any ideas?
>> ---------------- End original message. ---------------------
>>
>> It sounds to me like your system is waiting on disk resources (I am
>> assuming that is what the devfs state is), it is very likely trying
>> to use swap space.
>>
>> Now much memory does this system have?
>>
>> If it has insufficient memory to keep most things in RAM, it is
>> going to use a lot of swap and that will slow things down
>> considerably.
>>
>> Oh how I wish it were that simple...
>>
>> Mem: 1629M Active, 770M Inact, 392M Wired, 68K Cache, 214M Buf, 4791M
>> Free
>> Swap: 8192M Total, 8192M Free
>>
>> No swap in use, plenty of free memory.
>> ---------------- End original message. ---------------------
>>
>> Well, I took my best shot on the info presented. I still think there is
>> something in the disk I/O path that is acting as a bottleneck.
>>
>> Sorry it isn't so easy, I have nothing else here. Wish I could help.
>>
>> No worries, thanks for trying.
>>
>> I agree that it's probably disk-related, but other than these symptoms I
>> can't find anything wrong.
>>
>> I'll see if the FreeBSD list can help.
>>
>> -Stut
>>
>> --
>> http://stut.net/
>>
>>
>> ---------------------------------------------------------------------
>> The official User-To-User support forum of the Apache HTTP Server Project.
>> See <URL:http://httpd.apache.org/userslist.html> for more info.
>> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
>> " from the digest: users-digest-unsubscribe@httpd.apache.org
>> For additional commands, e-mail: users-help@httpd.apache.org
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> The official User-To-User support forum of the Apache HTTP Server Project.
>> See <URL:http://httpd.apache.org/userslist.html> for more info.
>> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
>> " from the digest: users-digest-unsubscribe@httpd.apache.org
>> For additional commands, e-mail: users-help@httpd.apache.org
>>
>>
>>
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>
Re: [users@httpd] High load, low CPU and processes in state "devfs"
Posted by Stut <st...@gmail.com>.
Thanks Mohit. Shortly after you sent this last night I enabled system
accounting. There's now a fair amount of data in the output but I'm
not really sure where to find the disk wait times...
root@harold:~# sa
41065 44351.99re 428.71cp 11avio 823k
8678 33375.24re 418.23cp 34avio 496k httpd*
2469 51.19re 4.79cp 2avio 767k convert
1513 1180.51re 2.69cp 48avio 4929k php
2469 56.01re 2.59cp 2avio 1312k composite
590 3945.74re 0.09cp 5avio 6182k smtp
624 152.45re 0.09cp 0avio 196079k perl*
46 468.10re 0.08cp 691avio 574k cleanup
1170 10.05re 0.08cp 11avio 107305k rateup
47 388.52re 0.03cp 190avio 1054k pickup
743 240.80re 0.02cp 0avio 27976k trivial-
rewrite
10443 1319.75re 0.00cp 0avio 26333351k sh
154 305.90re 0.00cp 24avio 44750k bounce
4454 101.21re 0.00cp 0avio 1753042k sendmail
4454 52.77re 0.00cp 7avio 2748826k postdrop
2 1.00re 0.00cp 0avio 3136k top
9 1278.02re 0.00cp 2avio 32944k ***other
389 13.73re 0.00cp 0avio 908587k ps
2 0.01re 0.00cp 6avio 981k sa
24 80.90re 0.00cp 0avio 60224k scache
9 35.12re 0.00cp 0avio 12700k anvil
78 0.28re 0.00cp 0avio 198900k atrun
623 1043.23re 0.00cp 0avio 959100k cron*
39 1.24re 0.00cp 2avio 112100k dd
27 52.78re 0.00cp 0avio 109600k error
65 114.24re 0.00cp 0avio 298000k flush
1165 41.51re 0.00cp 0avio 4475200k grep
40 0.14re 0.00cp 0avio 60000k jot
279 1.16re 0.00cp 3avio 971600k mv
7 0.31re 0.00cp 3avio 7200k newsyslog
13 1.68re 0.00cp 0avio 42500k proxymap
12 23.78re 0.00cp 0avio 78800k smtpd
40 0.16re 0.00cp 2avio 194800k unlink
388 14.47re 0.00cp 0avio 1219700k wc
Any help would be appreciated. Thanks.
-Stut
On 14 Jun 2008, at 00:43, Mohit Anchlia wrote:
> try "sa"
>
> On 6/13/08, Stut <st...@gmail.com> wrote: On 13 Jun 2008, at
> 23:38, Mohit Anchlia wrote:
> look at "sar" output and see the wait times on the disk is more than
> the service time.
>
> That command doesn't seem to exist on FreeBSD. Do you know if
> there's an equivalent command?
>
> -Stut
>
>
> On 6/13/08, Stut <st...@gmail.com> wrote:
>
> On 13 Jun 2008, at 22:56, Dragon wrote:
>
> Stut wrote:
> On 13 Jun 2008, at 22:37, Dragon wrote:
> Stut wrote:
> Hi,
>
> I have a problem with one of the web servers I manage. It runs
> FreeBSD
> 6.2, Apache 2.2.8 and PHP 5.2.1. It runs a high-ish number of pre-
> fork
> processes (usually around 240).
>
> What basically happens is that during our peak hours in the evening
> the site becomes very slow as does everything on the server. The top
> output shows nearly all httpd processes in the state "devfs" and the
> load is jumping up over 10 but the CPU is 70% idle. There's plenty of
> free memory, over 4GB, and lots of available disk space.
>
> While it's in this state all disk access is painfully slow. I've
> checked all the drives and the RAID card status and everything
> appears
> to be fine. It also recovers itself after a few hours when the
> traffic
> dies down again.
>
> I've spent a lot of today Googling but can't find any reference to
> this particular combination of symptoms. Does anyone have any ideas?
> ---------------- End original message. ---------------------
>
> It sounds to me like your system is waiting on disk resources (I am
> assuming that is what the devfs state is), it is very likely trying
> to use swap space.
>
> Now much memory does this system have?
>
> If it has insufficient memory to keep most things in RAM, it is
> going to use a lot of swap and that will slow things down
> considerably.
>
> Oh how I wish it were that simple...
>
> Mem: 1629M Active, 770M Inact, 392M Wired, 68K Cache, 214M Buf, 4791M
> Free
> Swap: 8192M Total, 8192M Free
>
> No swap in use, plenty of free memory.
> ---------------- End original message. ---------------------
>
> Well, I took my best shot on the info presented. I still think there
> is something in the disk I/O path that is acting as a bottleneck.
>
> Sorry it isn't so easy, I have nothing else here. Wish I could help.
>
> No worries, thanks for trying.
>
> I agree that it's probably disk-related, but other than these
> symptoms I can't find anything wrong.
>
> I'll see if the FreeBSD list can help.
>
> -Stut
>
> --
> http://stut.net/
>
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server
> Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>
>
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server
> Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] High load, low CPU and processes in state "devfs"
Posted by Mohit Anchlia <mo...@gmail.com>.
try "sa"
On 6/13/08, Stut <st...@gmail.com> wrote:
>
> On 13 Jun 2008, at 23:38, Mohit Anchlia wrote:
>
>> look at "sar" output and see the wait times on the disk is more than the
>> service time.
>>
>
> That command doesn't seem to exist on FreeBSD. Do you know if there's an
> equivalent command?
>
> -Stut
>
> On 6/13/08, Stut <st...@gmail.com> wrote:
>>
>> On 13 Jun 2008, at 22:56, Dragon wrote:
>>
>> Stut wrote:
>> On 13 Jun 2008, at 22:37, Dragon wrote:
>> Stut wrote:
>> Hi,
>>
>> I have a problem with one of the web servers I manage. It runs
>> FreeBSD
>> 6.2, Apache 2.2.8 and PHP 5.2.1. It runs a high-ish number of pre- fork
>> processes (usually around 240).
>>
>> What basically happens is that during our peak hours in the evening
>> the site becomes very slow as does everything on the server. The top
>> output shows nearly all httpd processes in the state "devfs" and the
>> load is jumping up over 10 but the CPU is 70% idle. There's plenty of
>> free memory, over 4GB, and lots of available disk space.
>>
>> While it's in this state all disk access is painfully slow. I've
>> checked all the drives and the RAID card status and everything
>> appears
>> to be fine. It also recovers itself after a few hours when the
>> traffic
>> dies down again.
>>
>> I've spent a lot of today Googling but can't find any reference to
>> this particular combination of symptoms. Does anyone have any ideas?
>> ---------------- End original message. ---------------------
>>
>> It sounds to me like your system is waiting on disk resources (I am
>> assuming that is what the devfs state is), it is very likely trying
>> to use swap space.
>>
>> Now much memory does this system have?
>>
>> If it has insufficient memory to keep most things in RAM, it is
>> going to use a lot of swap and that will slow things down
>> considerably.
>>
>> Oh how I wish it were that simple...
>>
>> Mem: 1629M Active, 770M Inact, 392M Wired, 68K Cache, 214M Buf, 4791M
>> Free
>> Swap: 8192M Total, 8192M Free
>>
>> No swap in use, plenty of free memory.
>> ---------------- End original message. ---------------------
>>
>> Well, I took my best shot on the info presented. I still think there is
>> something in the disk I/O path that is acting as a bottleneck.
>>
>> Sorry it isn't so easy, I have nothing else here. Wish I could help.
>>
>> No worries, thanks for trying.
>>
>> I agree that it's probably disk-related, but other than these symptoms I
>> can't find anything wrong.
>>
>> I'll see if the FreeBSD list can help.
>>
>> -Stut
>>
>> --
>> http://stut.net/
>>
>>
>> ---------------------------------------------------------------------
>> The official User-To-User support forum of the Apache HTTP Server Project.
>> See <URL:http://httpd.apache.org/userslist.html> for more info.
>> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
>> " from the digest: users-digest-unsubscribe@httpd.apache.org
>> For additional commands, e-mail: users-help@httpd.apache.org
>>
>>
>>
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>
Re: [users@httpd] High load, low CPU and processes in state "devfs"
Posted by Stut <st...@gmail.com>.
On 13 Jun 2008, at 23:38, Mohit Anchlia wrote:
> look at "sar" output and see the wait times on the disk is more than
> the service time.
That command doesn't seem to exist on FreeBSD. Do you know if there's
an equivalent command?
-Stut
> On 6/13/08, Stut <st...@gmail.com> wrote:
>
> On 13 Jun 2008, at 22:56, Dragon wrote:
>
> Stut wrote:
> On 13 Jun 2008, at 22:37, Dragon wrote:
> Stut wrote:
> Hi,
>
> I have a problem with one of the web servers I manage. It runs
> FreeBSD
> 6.2, Apache 2.2.8 and PHP 5.2.1. It runs a high-ish number of pre-
> fork
> processes (usually around 240).
>
> What basically happens is that during our peak hours in the evening
> the site becomes very slow as does everything on the server. The top
> output shows nearly all httpd processes in the state "devfs" and the
> load is jumping up over 10 but the CPU is 70% idle. There's plenty of
> free memory, over 4GB, and lots of available disk space.
>
> While it's in this state all disk access is painfully slow. I've
> checked all the drives and the RAID card status and everything
> appears
> to be fine. It also recovers itself after a few hours when the
> traffic
> dies down again.
>
> I've spent a lot of today Googling but can't find any reference to
> this particular combination of symptoms. Does anyone have any ideas?
> ---------------- End original message. ---------------------
>
> It sounds to me like your system is waiting on disk resources (I am
> assuming that is what the devfs state is), it is very likely trying
> to use swap space.
>
> Now much memory does this system have?
>
> If it has insufficient memory to keep most things in RAM, it is
> going to use a lot of swap and that will slow things down
> considerably.
>
> Oh how I wish it were that simple...
>
> Mem: 1629M Active, 770M Inact, 392M Wired, 68K Cache, 214M Buf, 4791M
> Free
> Swap: 8192M Total, 8192M Free
>
> No swap in use, plenty of free memory.
> ---------------- End original message. ---------------------
>
> Well, I took my best shot on the info presented. I still think there
> is something in the disk I/O path that is acting as a bottleneck.
>
> Sorry it isn't so easy, I have nothing else here. Wish I could help.
>
> No worries, thanks for trying.
>
> I agree that it's probably disk-related, but other than these
> symptoms I can't find anything wrong.
>
> I'll see if the FreeBSD list can help.
>
> -Stut
>
> --
> http://stut.net/
>
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server
> Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] High load, low CPU and processes in state "devfs"
Posted by Mohit Anchlia <mo...@gmail.com>.
look at "sar" output and see the wait times on the disk is more than the
service time.
On 6/13/08, Stut <st...@gmail.com> wrote:
>
>
> On 13 Jun 2008, at 22:56, Dragon wrote:
>
> Stut wrote:
>>
>>> On 13 Jun 2008, at 22:37, Dragon wrote:
>>>
>>>> Stut wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I have a problem with one of the web servers I manage. It runs
>>>>> FreeBSD
>>>>> 6.2, Apache 2.2.8 and PHP 5.2.1. It runs a high-ish number of pre-
>>>>> fork
>>>>> processes (usually around 240).
>>>>>
>>>>> What basically happens is that during our peak hours in the evening
>>>>> the site becomes very slow as does everything on the server. The top
>>>>> output shows nearly all httpd processes in the state "devfs" and the
>>>>> load is jumping up over 10 but the CPU is 70% idle. There's plenty of
>>>>> free memory, over 4GB, and lots of available disk space.
>>>>>
>>>>> While it's in this state all disk access is painfully slow. I've
>>>>> checked all the drives and the RAID card status and everything
>>>>> appears
>>>>> to be fine. It also recovers itself after a few hours when the
>>>>> traffic
>>>>> dies down again.
>>>>>
>>>>> I've spent a lot of today Googling but can't find any reference to
>>>>> this particular combination of symptoms. Does anyone have any ideas?
>>>>>
>>>> ---------------- End original message. ---------------------
>>>>
>>>> It sounds to me like your system is waiting on disk resources (I am
>>>> assuming that is what the devfs state is), it is very likely trying
>>>> to use swap space.
>>>>
>>>> Now much memory does this system have?
>>>>
>>>> If it has insufficient memory to keep most things in RAM, it is
>>>> going to use a lot of swap and that will slow things down
>>>> considerably.
>>>>
>>>
>>> Oh how I wish it were that simple...
>>>
>>> Mem: 1629M Active, 770M Inact, 392M Wired, 68K Cache, 214M Buf, 4791M
>>> Free
>>> Swap: 8192M Total, 8192M Free
>>>
>>> No swap in use, plenty of free memory.
>>>
>> ---------------- End original message. ---------------------
>>
>> Well, I took my best shot on the info presented. I still think there is
>> something in the disk I/O path that is acting as a bottleneck.
>>
>> Sorry it isn't so easy, I have nothing else here. Wish I could help.
>>
>
> No worries, thanks for trying.
>
> I agree that it's probably disk-related, but other than these symptoms I
> can't find anything wrong.
>
> I'll see if the FreeBSD list can help.
>
> -Stut
>
> --
> http://stut.net/
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>
Re: [users@httpd] High load, low CPU and processes in state "devfs"
Posted by Stut <st...@gmail.com>.
On 13 Jun 2008, at 22:56, Dragon wrote:
> Stut wrote:
>> On 13 Jun 2008, at 22:37, Dragon wrote:
>>> Stut wrote:
>>>> Hi,
>>>>
>>>> I have a problem with one of the web servers I manage. It runs
>>>> FreeBSD
>>>> 6.2, Apache 2.2.8 and PHP 5.2.1. It runs a high-ish number of
>>>> pre- fork
>>>> processes (usually around 240).
>>>>
>>>> What basically happens is that during our peak hours in the evening
>>>> the site becomes very slow as does everything on the server. The
>>>> top
>>>> output shows nearly all httpd processes in the state "devfs" and
>>>> the
>>>> load is jumping up over 10 but the CPU is 70% idle. There's
>>>> plenty of
>>>> free memory, over 4GB, and lots of available disk space.
>>>>
>>>> While it's in this state all disk access is painfully slow. I've
>>>> checked all the drives and the RAID card status and everything
>>>> appears
>>>> to be fine. It also recovers itself after a few hours when the
>>>> traffic
>>>> dies down again.
>>>>
>>>> I've spent a lot of today Googling but can't find any reference to
>>>> this particular combination of symptoms. Does anyone have any
>>>> ideas?
>>> ---------------- End original message. ---------------------
>>>
>>> It sounds to me like your system is waiting on disk resources (I am
>>> assuming that is what the devfs state is), it is very likely trying
>>> to use swap space.
>>>
>>> Now much memory does this system have?
>>>
>>> If it has insufficient memory to keep most things in RAM, it is
>>> going to use a lot of swap and that will slow things down
>>> considerably.
>>
>> Oh how I wish it were that simple...
>>
>> Mem: 1629M Active, 770M Inact, 392M Wired, 68K Cache, 214M Buf, 4791M
>> Free
>> Swap: 8192M Total, 8192M Free
>>
>> No swap in use, plenty of free memory.
> ---------------- End original message. ---------------------
>
> Well, I took my best shot on the info presented. I still think there
> is something in the disk I/O path that is acting as a bottleneck.
>
> Sorry it isn't so easy, I have nothing else here. Wish I could help.
No worries, thanks for trying.
I agree that it's probably disk-related, but other than these symptoms
I can't find anything wrong.
I'll see if the FreeBSD list can help.
-Stut
--
http://stut.net/
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] High load, low CPU and processes in state
"devfs"
Posted by Dragon <dr...@crimson-dragon.com>.
Stut wrote:
>On 13 Jun 2008, at 22:37, Dragon wrote:
>>Stut wrote:
>>>Hi,
>>>
>>>I have a problem with one of the web servers I manage. It runs
>>>FreeBSD
>>>6.2, Apache 2.2.8 and PHP 5.2.1. It runs a high-ish number of pre- fork
>>>processes (usually around 240).
>>>
>>>What basically happens is that during our peak hours in the evening
>>>the site becomes very slow as does everything on the server. The top
>>>output shows nearly all httpd processes in the state "devfs" and the
>>>load is jumping up over 10 but the CPU is 70% idle. There's plenty of
>>>free memory, over 4GB, and lots of available disk space.
>>>
>>>While it's in this state all disk access is painfully slow. I've
>>>checked all the drives and the RAID card status and everything
>>>appears
>>>to be fine. It also recovers itself after a few hours when the
>>>traffic
>>>dies down again.
>>>
>>>I've spent a lot of today Googling but can't find any reference to
>>>this particular combination of symptoms. Does anyone have any ideas?
>>---------------- End original message. ---------------------
>>
>>It sounds to me like your system is waiting on disk resources (I am
>>assuming that is what the devfs state is), it is very likely trying
>>to use swap space.
>>
>>Now much memory does this system have?
>>
>>If it has insufficient memory to keep most things in RAM, it is
>>going to use a lot of swap and that will slow things down
>>considerably.
>
>Oh how I wish it were that simple...
>
>Mem: 1629M Active, 770M Inact, 392M Wired, 68K Cache, 214M Buf, 4791M
>Free
>Swap: 8192M Total, 8192M Free
>
>No swap in use, plenty of free memory.
---------------- End original message. ---------------------
Well, I took my best shot on the info presented. I still think there
is something in the disk I/O path that is acting as a bottleneck.
Sorry it isn't so easy, I have nothing else here. Wish I could help.
Dragon
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Venimus, Saltavimus, Bibimus (et naribus canium capti sumus)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] High load, low CPU and processes in state "devfs"
Posted by Stut <st...@gmail.com>.
On 13 Jun 2008, at 22:37, Dragon wrote:
> Stut wrote:
>> Hi,
>>
>> I have a problem with one of the web servers I manage. It runs
>> FreeBSD
>> 6.2, Apache 2.2.8 and PHP 5.2.1. It runs a high-ish number of pre-
>> fork
>> processes (usually around 240).
>>
>> What basically happens is that during our peak hours in the evening
>> the site becomes very slow as does everything on the server. The top
>> output shows nearly all httpd processes in the state "devfs" and the
>> load is jumping up over 10 but the CPU is 70% idle. There's plenty of
>> free memory, over 4GB, and lots of available disk space.
>>
>> While it's in this state all disk access is painfully slow. I've
>> checked all the drives and the RAID card status and everything
>> appears
>> to be fine. It also recovers itself after a few hours when the
>> traffic
>> dies down again.
>>
>> I've spent a lot of today Googling but can't find any reference to
>> this particular combination of symptoms. Does anyone have any ideas?
> ---------------- End original message. ---------------------
>
> It sounds to me like your system is waiting on disk resources (I am
> assuming that is what the devfs state is), it is very likely trying
> to use swap space.
>
> Now much memory does this system have?
>
> If it has insufficient memory to keep most things in RAM, it is
> going to use a lot of swap and that will slow things down
> considerably.
Oh how I wish it were that simple...
Mem: 1629M Active, 770M Inact, 392M Wired, 68K Cache, 214M Buf, 4791M
Free
Swap: 8192M Total, 8192M Free
No swap in use, plenty of free memory.
-Stut
--
http://stut.net/
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] High load, low CPU and processes in state
"devfs"
Posted by Dragon <dr...@crimson-dragon.com>.
Stut wrote:
>Hi,
>
>I have a problem with one of the web servers I manage. It runs FreeBSD
>6.2, Apache 2.2.8 and PHP 5.2.1. It runs a high-ish number of pre-fork
>processes (usually around 240).
>
>What basically happens is that during our peak hours in the evening
>the site becomes very slow as does everything on the server. The top
>output shows nearly all httpd processes in the state "devfs" and the
>load is jumping up over 10 but the CPU is 70% idle. There's plenty of
>free memory, over 4GB, and lots of available disk space.
>
>While it's in this state all disk access is painfully slow. I've
>checked all the drives and the RAID card status and everything appears
>to be fine. It also recovers itself after a few hours when the traffic
>dies down again.
>
>I've spent a lot of today Googling but can't find any reference to
>this particular combination of symptoms. Does anyone have any ideas?
---------------- End original message. ---------------------
It sounds to me like your system is waiting on disk resources (I am
assuming that is what the devfs state is), it is very likely trying
to use swap space.
Now much memory does this system have?
If it has insufficient memory to keep most things in RAM, it is going
to use a lot of swap and that will slow things down considerably.
Dragon
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Venimus, Saltavimus, Bibimus (et naribus canium capti sumus)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org