You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@httpd.apache.org by Stuart Haas <sh...@vibespin.com> on 2004/06/16 18:49:37 UTC
[users@httpd] Server Load Issue
Hi,
We are having a strange problem with Apache where the server load
skyrockets and leaves our site very unresponsive. The load average on
the server typically is well under 0.5. Once a threshold is reached (I
can't pin this down exactly, but we usually reach it several times over
our busy hours), the load immediately jumps to around 150 and more.
After a while, every thing returns to normal. I would expect that the
server load would increase linearly as traffic increases, but that is
not the case. Out site is not that busy as we serve about 30,000 pages
a day.
We are running apache 2.0.46 (Redhat's) on Redhat Enterprise Linux 3.
The hardware is 2x 3.06Ghz Xeon with 4GB RAM, so I don't believe the
box is underpowered. Our content management system is PHP based and in
an effort to solve this issue went to the Zend performance suite, but
this has not had any effect on this problem. There are no error
messages in any of the logs.
How would we go about troubleshooting this issue?
Stuart
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Server Load Issue
Posted by Stuart Haas <sh...@vibespin.com>.
So, it looks like I have hit a dead end here. Can any one suggest a
better forum for trying to get this debugged?
Stuart
Stuart Haas wrote:
> No, we don't run out of memory when the threshold it hit, it only
> happens once the load gets well above 100. Even then, we don't use
> that much swap.
>
> Joshua Slive wrote:
>
>>
>> On Thu, 17 Jun 2004, Stuart Haas wrote:
>>
>>>
>>> Here we go - the load average was at aprox 350 and coming down.
>>
>>
>>
>> Where you out of memory at the time? Was the server thrashing in
>> swap? You may need to decrease MaxClients.
>>
>> Joshua.
>>
>> ---------------------------------------------------------------------
>> The official User-To-User support forum of the Apache HTTP Server
>> Project.
>> See <URL:http://httpd.apache.org/userslist.html> for more info.
>> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
>> " from the digest: users-digest-unsubscribe@httpd.apache.org
>> For additional commands, e-mail: users-help@httpd.apache.org
>>
>
--
------------------------------
Stuart Haas
IT Director
Vibe/Spin Ventures
---
Assistant Director of Technology
Miller Publishing Group
212.448.7493
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Server Load Issue
Posted by Stuart Haas <sh...@vibespin.com>.
No, we don't run out of memory when the threshold it hit, it only
happens once the load gets well above 100. Even then, we don't use that
much swap.
Joshua Slive wrote:
>
> On Thu, 17 Jun 2004, Stuart Haas wrote:
>
>>
>> Here we go - the load average was at aprox 350 and coming down.
>
>
> Where you out of memory at the time? Was the server thrashing in
> swap? You may need to decrease MaxClients.
>
> Joshua.
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server
> Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
--
------------------------------
Stuart Haas
IT Director
Vibe/Spin Ventures
---
Assistant Director of Technology
Miller Publishing Group
212.448.7493
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Server Load Issue
Posted by Joshua Slive <jo...@slive.ca>.
On Thu, 17 Jun 2004, Stuart Haas wrote:
>
> Here we go - the load average was at aprox 350 and coming down.
Where you out of memory at the time? Was the server thrashing in swap?
You may need to decrease MaxClients.
Joshua.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Server Load Issue
Posted by Andrea Cerrito <is...@gentestrana.net>.
Il gio, 2004-06-17 alle 11:58, Robert Andersson ha scritto:
> Stuart Haas wrote:
> > This is what we get: (server status and a whole lot more...):
> >
> > <snip>
> > WWWWWWWWWWWWWWWWWWWWWWWWW_
>
> This looks familiar to a problem I have had from time to time, but I have
> not been able to reproduce it.
>
> What seems to happen is that suddenly requests from a set of clients
> "hangs", they just keep being in working state and no response is sent
> (checked with a sniffer). If the same client makes other requests (including
> to other vhosts) they will hang as well, but other clients can still fetch
> them fine. After a while all goes back to normal.
>
> I'm not 100% sure that this is triggered by increased load, but it leans
> towards that. However, high load does not always cause this behavior.
>
> I've been meaning to investigate this more, and see if I could determine the
> cause, but I've had no time. I have only seen this on Windows, and had
> thought it was a Windows issue :(
I found interesting even the higher values of iowait, that can be a
consequence of the problem reported by Robert.
Please report a ps of your machine, and note strange behaviours of httpd
processes.
--
Enjoy your freedom
Andrea Cerrito
http://www.gentestrana.net/
Linux User #103564
=== (17:57:49) Nietzsche: "niente è quello che sembra"
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Server Load Issue
Posted by Robert Andersson <ro...@profundis.nu>.
Stuart Haas wrote:
> This is what we get: (server status and a whole lot more...):
>
> <snip>
> WWWWWWWWWWWWWWWWWWWWWWWWW_
This looks familiar to a problem I have had from time to time, but I have
not been able to reproduce it.
What seems to happen is that suddenly requests from a set of clients
"hangs", they just keep being in working state and no response is sent
(checked with a sniffer). If the same client makes other requests (including
to other vhosts) they will hang as well, but other clients can still fetch
them fine. After a while all goes back to normal.
I'm not 100% sure that this is triggered by increased load, but it leans
towards that. However, high load does not always cause this behavior.
I've been meaning to investigate this more, and see if I could determine the
cause, but I've had no time. I have only seen this on Windows, and had
thought it was a Windows issue :(
Regards,
Robert Andersson
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Server Load Issue
Posted by Stuart Haas <sh...@vibespin.com>.
I don't don't think it's a DOS. The W's appear as a result of the load,
not as the cause of it. I was watching as the load build and everything
went to W. Hostnamelookups are off.
Server status with a normal load:
Server Version: Apache
Server Built: Mar 1 2004 12:14:32
------------------------------------------------------------------------
Current Time: Wednesday, 16-Jun-2004 14:41:38 EDT
Restart Time: Tuesday, 15-Jun-2004 14:56:46 EDT
Parent Server Generation: 0
Server uptime: 23 hours 44 minutes 51 seconds
Total accesses: 740073 - Total Traffic: 6.7 GB
CPU Usage: u1034.06 s33.86 cu2.35 cs1.93 - 1.25% CPU load
8.66 requests/sec - 81.8 kB/second - 9.5 kB/request
34 requests currently being processed, 30 idle workers
WK_K__KK___W_KK______KK_WW_KK___.K_CKK_KKK_K__K_W_WK.W_K_KK.W..W
......_...................K._...................................
..............._.................................W..............
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
........................................................
Vasiliy Boulytchev wrote:
>Ok, Please tell me if I am wrong, but "W" state is "sending reply" hmmmmmm.
>DOS attack? Does your web server do DNS queries? If so, turn them
>off....... Anyone else?
>
>I assume no other procs are thrashing the system.........
>
>
>Vasiliy Boulytchev
>Colorado Information Technologies, Inc.
>http://www.coinfotech.com
>
>-----Original Message-----
>From: Stuart Haas [mailto:shaas@vibespin.com]
>Sent: Wednesday, June 16, 2004 12:25 PM
>To: users@httpd.apache.org
>Subject: Re: [users@httpd] Server Load Issue
>
>
>Ok,
>
>This is what we get: (server status and a whole lot more...):
>
>Server Version: Apache
>Server Built: Mar 1 2004 12:14:32
>
>Current Time: Wednesday, 16-Jun-2004 14:12:27 EDT Restart Time: Tuesday,
>15-Jun-2004 14:56:46 EDT Parent Server Generation: 0 Server uptime: 23 hours
>15 minutes 41 seconds Total accesses: 717294 - Total Traffic: 6.5 GB CPU
>Usage: u382.63 s15.45 cu1.57 cs1.27 - .479% CPU load
>8.57 requests/sec - 80.9 kB/second - 9.4 kB/request
>308 requests currently being processed, 8 idle workers
>
>WWWWWWWWWWWWWWWWWWWWWWWWWW_WWWWWWWWWWWWWWWWWWWWWWWWWWWWWW_WWWWWW
>WWWWWW_WWWWWWWWWWWWWWWWWWWWW_WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
>WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
>WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWKWWWWWWWWWWWWW
>WWRWWWWWWWWWWWWWWWWWWWWWWWWW_WWWWWW_WWWWWWWWWKWW_WWWWWW_WWWW....
>................................................................
>................................................................
>................................................................
>................................................................
>................................................................
>................................................................
>................................................................
>................................................................
>................................................................
>................................................................
>................................................................
>................................................................
>................................................................
>................................................................
>................................................................
>................................................................
>........................................................
>
>
>---
>
>( I think this is when things are cooling down - CPU hits 100% on all 4 when
>peaking)
>
>Top:
>14:12:30 up 27 days, 13:00, 4 users, load average: 261.00, 133.84, 55.70
>391 processes: 370 sleeping, 19 running, 2 zombie, 0 stopped CPU states: cpu
>user nice system irq softirq iowait idle total 42.7% 0.0% 3.5% 0.1% 0.5%
>52.9% 0.0% cpu00 37.6% 0.0% 4.0% 0.0% 1.1% 57.1% 0.0%
>cpu01 35.2% 0.0% 1.7% 0.6% 0.0% 62.3% 0.0%
>cpu02 61.8% 0.0% 4.9% 0.0% 0.8% 32.3% 0.0%
>cpu03 36.1% 0.0% 3.5% 0.0% 0.3% 59.9% 0.0%
>Mem: 4125972k av, 4105812k used, 20160k free, 0k shrd, 3848k buff 3213112k
>actv, 612800k in_d, 63032k in_c
>Swap: 4096020k av, 867116k used, 3228904k free 865776k cached
>
>---
>
>iostat 5
>
>avg-cpu: %user %nice %sys %idle
>5.45 0.00 0.24 94.32
>
>Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn dev3-0 4.36 22.42 106.24
>52403004 248317390
>
>avg-cpu: %user %nice %sys %idle
>79.79 0.00 20.21 0.00
>
>Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn dev3-0 47.83 347.83
>1565.22 80 360
>
>avg-cpu: %user %nice %sys %idle
>95.02 0.00 4.98 0.00
>
>Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn dev3-0 38.00 688.00
>272.00 344 136
>
>avg-cpu: %user %nice %sys %idle
>59.15 0.00 40.85 0.00
>
>Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn dev3-0 145.71 937.14
>1165.71 328 408
>
>avg-cpu: %user %nice %sys %idle
>94.86 0.00 5.14 0.00
>
>Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn dev3-0 15.87 0.00 660.32
>0 416
>
>avg-cpu: %user %nice %sys %idle
>96.95 0.00 3.05 0.00
>
>Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn dev3-0 71.23 0.00 898.63
>0 656
>
>avg-cpu: %user %nice %sys %idle
>92.92 0.00 7.08 0.00
>---
>
>vmstat 5
>procs memory swap io system cpu
>r b swpd free buff cache si so bi bo in cs us sy id wa 0 328 1019384 23668
>3380 725624 3 0 4 0 2 2 1 0 2 2 0 328 1019828 24896 3396 725500 442 134 442
>150 277 493 2 1 0 97 0 330 1019468 29936 3412 725528 466 0 466 11 282 326 2
>0 0 98 0 324 1019660 27316 3444 725656 478 124 483 131 298 359 3 1 0 96 0
>320 1015860 70088 3460 725676 428 308 429 330 308 367 3 1 0 96 0 323 1015500
>71708 3476 725696 453 0 453 13 452 521 15 1 0 84 0 326 1014520 90796 3488
>725716 413 54 414 119 413 468 13 1 0 86
>11 289 1012908 75644 3524 725760 517 0 522 18 1221 692 61 4 0 35 0 304
>1012472 76696 3540 725856 430 142 444 199 881 684 49 2 0 49 0 302 1012052
>78696 3568 725880 438 0 438 50 393 411 9 0 0 91
>64 176 1005584 72120 3852 725972 478 46 537 78 1540 1063 69 4 0 27
>---
>
>pstree
>init-T-atd
>+-bdflush
>+-cache_clean
>+-crond---crond---cache_clean
>+-gpm
>+-httpd-T-305*[httpd]
>| L-4*[httpd---links]
>+-irqbalance
>+-keventd
>+-khubd
>+-3*[kjournald]
>+-klogd
>+-kscand
>+-ksoftirqd/0
>+-ksoftirqd/1
>+-ksoftirqd/2
>+-ksoftirqd/3
>+-kswapd
>+-kupdated
>+-mdadm
>+-mdmpd
>+-mdrecoveryd
>+-6*[mingetty]
>+-ntpd
>+-3*[raid1d]
>+-rhnsd
>+-2*[sendmail]
>+-sldfs_rreq
>+-sldrm_coll---sldrm_hist
>+-sldtkd
>+-sshd-T-sshd---sshd
>| +-sshd---sshd---bash---top
>| +-sshd---sshd---bash---su---bash---top
>| +-sshd---sshd---bash
>| L-sshd---sshd---bash---su---bash---pstree
>+-syslogd
>L-xinetd
>
>Vasiliy Boulytchev wrote:
>
>
>
>>During the slowdown, what does /server-status/ show?
>>
>>
>>Vasiliy Boulytchev
>>Colorado Information Technologies, Inc.
>>http://www.coinfotech.com
>>
>>-----Original Message-----
>>From: Stuart Haas [mailto:shaas@vibespin.com]
>>Sent: Wednesday, June 16, 2004 10:50 AM
>>To: users@httpd.apache.org
>>Subject: [users@httpd] Server Load Issue
>>
>>
>>Hi,
>>
>>We are having a strange problem with Apache where the server load
>>skyrockets and leaves our site very unresponsive. The load average on
>>the server typically is well under 0.5. Once a threshold is reached (I
>>can't pin this down exactly, but we usually reach it several times over
>>our busy hours), the load immediately jumps to around 150 and more.
>>After a while, every thing returns to normal. I would expect that the
>>server load would increase linearly as traffic increases, but that is
>>not the case. Out site is not that busy as we serve about 30,000 pages
>>a day.
>>
>>We are running apache 2.0.46 (Redhat's) on Redhat Enterprise Linux 3.
>>The hardware is 2x 3.06Ghz Xeon with 4GB RAM, so I don't believe the
>>box is underpowered. Our content management system is PHP based and in
>>an effort to solve this issue went to the Zend performance suite, but
>>this has not had any effect on this problem. There are no error messages
>>
>>
>in any of the logs.
>
>
>>How would we go about troubleshooting this issue?
>>
>>Stuart
>>
>>
>>
>>---------------------------------------------------------------------
>>The official User-To-User support forum of the Apache HTTP Server Project.
>>See <URL:http://httpd.apache.org/userslist.html> for more info.
>>To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
>> " from the digest: users-digest-unsubscribe@httpd.apache.org
>>For additional commands, e-mail: users-help@httpd.apache.org
>>
>>
>>
>>---------------------------------------------------------------------
>>The official User-To-User support forum of the Apache HTTP Server Project.
>>See <URL:http://httpd.apache.org/userslist.html> for more info.
>>To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
>> " from the digest: users-digest-unsubscribe@httpd.apache.org
>>For additional commands, e-mail: users-help@httpd.apache.org
>>
>>
>>
>>
>>
>
>--
>------------------------------
>Stuart Haas
>IT Director
>Vibe/Spin Ventures
>---
>Assistant Director of Technology
>Miller Publishing Group
>212.448.7493
>
>
>
>---------------------------------------------------------------------
>The official User-To-User support forum of the Apache HTTP Server Project.
>See <URL:http://httpd.apache.org/userslist.html> for more info.
>To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
>For additional commands, e-mail: users-help@httpd.apache.org
>
>
>
>---------------------------------------------------------------------
>The official User-To-User support forum of the Apache HTTP Server Project.
>See <URL:http://httpd.apache.org/userslist.html> for more info.
>To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
>For additional commands, e-mail: users-help@httpd.apache.org
>
>
>
--
------------------------------
Stuart Haas
IT Director
Vibe/Spin Ventures
---
Assistant Director of Technology
Miller Publishing Group
212.448.7493
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
RE: [users@httpd] Server Load Issue
Posted by Vasiliy Boulytchev <va...@boulytcheva.com>.
Ok, Please tell me if I am wrong, but "W" state is "sending reply" hmmmmmm.
DOS attack? Does your web server do DNS queries? If so, turn them
off....... Anyone else?
I assume no other procs are thrashing the system.........
Vasiliy Boulytchev
Colorado Information Technologies, Inc.
http://www.coinfotech.com
-----Original Message-----
From: Stuart Haas [mailto:shaas@vibespin.com]
Sent: Wednesday, June 16, 2004 12:25 PM
To: users@httpd.apache.org
Subject: Re: [users@httpd] Server Load Issue
Ok,
This is what we get: (server status and a whole lot more...):
Server Version: Apache
Server Built: Mar 1 2004 12:14:32
Current Time: Wednesday, 16-Jun-2004 14:12:27 EDT Restart Time: Tuesday,
15-Jun-2004 14:56:46 EDT Parent Server Generation: 0 Server uptime: 23 hours
15 minutes 41 seconds Total accesses: 717294 - Total Traffic: 6.5 GB CPU
Usage: u382.63 s15.45 cu1.57 cs1.27 - .479% CPU load
8.57 requests/sec - 80.9 kB/second - 9.4 kB/request
308 requests currently being processed, 8 idle workers
WWWWWWWWWWWWWWWWWWWWWWWWWW_WWWWWWWWWWWWWWWWWWWWWWWWWWWWWW_WWWWWW
WWWWWW_WWWWWWWWWWWWWWWWWWWWW_WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWKWWWWWWWWWWWWW
WWRWWWWWWWWWWWWWWWWWWWWWWWWW_WWWWWW_WWWWWWWWWKWW_WWWWWW_WWWW....
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
........................................................
---
( I think this is when things are cooling down - CPU hits 100% on all 4 when
peaking)
Top:
14:12:30 up 27 days, 13:00, 4 users, load average: 261.00, 133.84, 55.70
391 processes: 370 sleeping, 19 running, 2 zombie, 0 stopped CPU states: cpu
user nice system irq softirq iowait idle total 42.7% 0.0% 3.5% 0.1% 0.5%
52.9% 0.0% cpu00 37.6% 0.0% 4.0% 0.0% 1.1% 57.1% 0.0%
cpu01 35.2% 0.0% 1.7% 0.6% 0.0% 62.3% 0.0%
cpu02 61.8% 0.0% 4.9% 0.0% 0.8% 32.3% 0.0%
cpu03 36.1% 0.0% 3.5% 0.0% 0.3% 59.9% 0.0%
Mem: 4125972k av, 4105812k used, 20160k free, 0k shrd, 3848k buff 3213112k
actv, 612800k in_d, 63032k in_c
Swap: 4096020k av, 867116k used, 3228904k free 865776k cached
---
iostat 5
avg-cpu: %user %nice %sys %idle
5.45 0.00 0.24 94.32
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn dev3-0 4.36 22.42 106.24
52403004 248317390
avg-cpu: %user %nice %sys %idle
79.79 0.00 20.21 0.00
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn dev3-0 47.83 347.83
1565.22 80 360
avg-cpu: %user %nice %sys %idle
95.02 0.00 4.98 0.00
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn dev3-0 38.00 688.00
272.00 344 136
avg-cpu: %user %nice %sys %idle
59.15 0.00 40.85 0.00
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn dev3-0 145.71 937.14
1165.71 328 408
avg-cpu: %user %nice %sys %idle
94.86 0.00 5.14 0.00
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn dev3-0 15.87 0.00 660.32
0 416
avg-cpu: %user %nice %sys %idle
96.95 0.00 3.05 0.00
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn dev3-0 71.23 0.00 898.63
0 656
avg-cpu: %user %nice %sys %idle
92.92 0.00 7.08 0.00
---
vmstat 5
procs memory swap io system cpu
r b swpd free buff cache si so bi bo in cs us sy id wa 0 328 1019384 23668
3380 725624 3 0 4 0 2 2 1 0 2 2 0 328 1019828 24896 3396 725500 442 134 442
150 277 493 2 1 0 97 0 330 1019468 29936 3412 725528 466 0 466 11 282 326 2
0 0 98 0 324 1019660 27316 3444 725656 478 124 483 131 298 359 3 1 0 96 0
320 1015860 70088 3460 725676 428 308 429 330 308 367 3 1 0 96 0 323 1015500
71708 3476 725696 453 0 453 13 452 521 15 1 0 84 0 326 1014520 90796 3488
725716 413 54 414 119 413 468 13 1 0 86
11 289 1012908 75644 3524 725760 517 0 522 18 1221 692 61 4 0 35 0 304
1012472 76696 3540 725856 430 142 444 199 881 684 49 2 0 49 0 302 1012052
78696 3568 725880 438 0 438 50 393 411 9 0 0 91
64 176 1005584 72120 3852 725972 478 46 537 78 1540 1063 69 4 0 27
---
pstree
init-T-atd
+-bdflush
+-cache_clean
+-crond---crond---cache_clean
+-gpm
+-httpd-T-305*[httpd]
| L-4*[httpd---links]
+-irqbalance
+-keventd
+-khubd
+-3*[kjournald]
+-klogd
+-kscand
+-ksoftirqd/0
+-ksoftirqd/1
+-ksoftirqd/2
+-ksoftirqd/3
+-kswapd
+-kupdated
+-mdadm
+-mdmpd
+-mdrecoveryd
+-6*[mingetty]
+-ntpd
+-3*[raid1d]
+-rhnsd
+-2*[sendmail]
+-sldfs_rreq
+-sldrm_coll---sldrm_hist
+-sldtkd
+-sshd-T-sshd---sshd
| +-sshd---sshd---bash---top
| +-sshd---sshd---bash---su---bash---top
| +-sshd---sshd---bash
| L-sshd---sshd---bash---su---bash---pstree
+-syslogd
L-xinetd
Vasiliy Boulytchev wrote:
>During the slowdown, what does /server-status/ show?
>
>
>Vasiliy Boulytchev
>Colorado Information Technologies, Inc.
>http://www.coinfotech.com
>
>-----Original Message-----
>From: Stuart Haas [mailto:shaas@vibespin.com]
>Sent: Wednesday, June 16, 2004 10:50 AM
>To: users@httpd.apache.org
>Subject: [users@httpd] Server Load Issue
>
>
>Hi,
>
>We are having a strange problem with Apache where the server load
>skyrockets and leaves our site very unresponsive. The load average on
>the server typically is well under 0.5. Once a threshold is reached (I
>can't pin this down exactly, but we usually reach it several times over
>our busy hours), the load immediately jumps to around 150 and more.
>After a while, every thing returns to normal. I would expect that the
>server load would increase linearly as traffic increases, but that is
>not the case. Out site is not that busy as we serve about 30,000 pages
>a day.
>
>We are running apache 2.0.46 (Redhat's) on Redhat Enterprise Linux 3.
>The hardware is 2x 3.06Ghz Xeon with 4GB RAM, so I don't believe the
>box is underpowered. Our content management system is PHP based and in
>an effort to solve this issue went to the Zend performance suite, but
>this has not had any effect on this problem. There are no error messages
in any of the logs.
>
>How would we go about troubleshooting this issue?
>
>Stuart
>
>
>
>---------------------------------------------------------------------
>The official User-To-User support forum of the Apache HTTP Server Project.
>See <URL:http://httpd.apache.org/userslist.html> for more info.
>To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
>For additional commands, e-mail: users-help@httpd.apache.org
>
>
>
>---------------------------------------------------------------------
>The official User-To-User support forum of the Apache HTTP Server Project.
>See <URL:http://httpd.apache.org/userslist.html> for more info.
>To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
>For additional commands, e-mail: users-help@httpd.apache.org
>
>
>
--
------------------------------
Stuart Haas
IT Director
Vibe/Spin Ventures
---
Assistant Director of Technology
Miller Publishing Group
212.448.7493
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Server Load Issue
Posted by Stuart Haas <sh...@vibespin.com>.
Ok,
This is what we get: (server status and a whole lot more...):
Server Version: Apache
Server Built: Mar 1 2004 12:14:32
Current Time: Wednesday, 16-Jun-2004 14:12:27 EDT
Restart Time: Tuesday, 15-Jun-2004 14:56:46 EDT
Parent Server Generation: 0
Server uptime: 23 hours 15 minutes 41 seconds
Total accesses: 717294 - Total Traffic: 6.5 GB
CPU Usage: u382.63 s15.45 cu1.57 cs1.27 - .479% CPU load
8.57 requests/sec - 80.9 kB/second - 9.4 kB/request
308 requests currently being processed, 8 idle workers
WWWWWWWWWWWWWWWWWWWWWWWWWW_WWWWWWWWWWWWWWWWWWWWWWWWWWWWWW_WWWWWW
WWWWWW_WWWWWWWWWWWWWWWWWWWWW_WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWKWWWWWWWWWWWWW
WWRWWWWWWWWWWWWWWWWWWWWWWWWW_WWWWWW_WWWWWWWWWKWW_WWWWWW_WWWW....
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
........................................................
---
( I think this is when things are cooling down - CPU hits 100% on all 4
when peaking)
Top:
14:12:30 up 27 days, 13:00, 4 users, load average: 261.00, 133.84, 55.70
391 processes: 370 sleeping, 19 running, 2 zombie, 0 stopped
CPU states: cpu user nice system irq softirq iowait idle
total 42.7% 0.0% 3.5% 0.1% 0.5% 52.9% 0.0%
cpu00 37.6% 0.0% 4.0% 0.0% 1.1% 57.1% 0.0%
cpu01 35.2% 0.0% 1.7% 0.6% 0.0% 62.3% 0.0%
cpu02 61.8% 0.0% 4.9% 0.0% 0.8% 32.3% 0.0%
cpu03 36.1% 0.0% 3.5% 0.0% 0.3% 59.9% 0.0%
Mem: 4125972k av, 4105812k used, 20160k free, 0k shrd, 3848k buff
3213112k actv, 612800k in_d, 63032k in_c
Swap: 4096020k av, 867116k used, 3228904k free 865776k cached
---
iostat 5
avg-cpu: %user %nice %sys %idle
5.45 0.00 0.24 94.32
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
dev3-0 4.36 22.42 106.24 52403004 248317390
avg-cpu: %user %nice %sys %idle
79.79 0.00 20.21 0.00
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
dev3-0 47.83 347.83 1565.22 80 360
avg-cpu: %user %nice %sys %idle
95.02 0.00 4.98 0.00
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
dev3-0 38.00 688.00 272.00 344 136
avg-cpu: %user %nice %sys %idle
59.15 0.00 40.85 0.00
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
dev3-0 145.71 937.14 1165.71 328 408
avg-cpu: %user %nice %sys %idle
94.86 0.00 5.14 0.00
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
dev3-0 15.87 0.00 660.32 0 416
avg-cpu: %user %nice %sys %idle
96.95 0.00 3.05 0.00
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
dev3-0 71.23 0.00 898.63 0 656
avg-cpu: %user %nice %sys %idle
92.92 0.00 7.08 0.00
---
vmstat 5
procs memory swap io system cpu
r b swpd free buff cache si so bi bo in cs us sy id wa
0 328 1019384 23668 3380 725624 3 0 4 0 2 2 1 0 2 2
0 328 1019828 24896 3396 725500 442 134 442 150 277 493 2 1 0 97
0 330 1019468 29936 3412 725528 466 0 466 11 282 326 2 0 0 98
0 324 1019660 27316 3444 725656 478 124 483 131 298 359 3 1 0 96
0 320 1015860 70088 3460 725676 428 308 429 330 308 367 3 1 0 96
0 323 1015500 71708 3476 725696 453 0 453 13 452 521 15 1 0 84
0 326 1014520 90796 3488 725716 413 54 414 119 413 468 13 1 0 86
11 289 1012908 75644 3524 725760 517 0 522 18 1221 692 61 4 0 35
0 304 1012472 76696 3540 725856 430 142 444 199 881 684 49 2 0 49
0 302 1012052 78696 3568 725880 438 0 438 50 393 411 9 0 0 91
64 176 1005584 72120 3852 725972 478 46 537 78 1540 1063 69 4 0 27
---
pstree
init─┬─atd
├─bdflush
├─cache_clean
├─crond───crond───cache_clean
├─gpm
├─httpd─┬─305*[httpd]
│ └─4*[httpd───links]
├─irqbalance
├─keventd
├─khubd
├─3*[kjournald]
├─klogd
├─kscand
├─ksoftirqd/0
├─ksoftirqd/1
├─ksoftirqd/2
├─ksoftirqd/3
├─kswapd
├─kupdated
├─mdadm
├─mdmpd
├─mdrecoveryd
├─6*[mingetty]
├─ntpd
├─3*[raid1d]
├─rhnsd
├─2*[sendmail]
├─sldfs_rreq
├─sldrm_coll───sldrm_hist
├─sldtkd
├─sshd─┬─sshd───sshd
│ ├─sshd───sshd───bash───top
│ ├─sshd───sshd───bash───su───bash───top
│ ├─sshd───sshd───bash
│ └─sshd───sshd───bash───su───bash───pstree
├─syslogd
└─xinetd
Vasiliy Boulytchev wrote:
>During the slowdown, what does /server-status/ show?
>
>
>Vasiliy Boulytchev
>Colorado Information Technologies, Inc.
>http://www.coinfotech.com
>
>-----Original Message-----
>From: Stuart Haas [mailto:shaas@vibespin.com]
>Sent: Wednesday, June 16, 2004 10:50 AM
>To: users@httpd.apache.org
>Subject: [users@httpd] Server Load Issue
>
>
>Hi,
>
>We are having a strange problem with Apache where the server load skyrockets
>and leaves our site very unresponsive. The load average on the server
>typically is well under 0.5. Once a threshold is reached (I can't pin this
>down exactly, but we usually reach it several times over our busy hours),
>the load immediately jumps to around 150 and more.
>After a while, every thing returns to normal. I would expect that the
>server load would increase linearly as traffic increases, but that is
>not the case. Out site is not that busy as we serve about 30,000 pages
>a day.
>
>We are running apache 2.0.46 (Redhat's) on Redhat Enterprise Linux 3.
>The hardware is 2x 3.06Ghz Xeon with 4GB RAM, so I don't believe the box is
>underpowered. Our content management system is PHP based and in an effort
>to solve this issue went to the Zend performance suite, but this has not had
>any effect on this problem. There are no error messages in any of the logs.
>
>How would we go about troubleshooting this issue?
>
>Stuart
>
>
>
>---------------------------------------------------------------------
>The official User-To-User support forum of the Apache HTTP Server Project.
>See <URL:http://httpd.apache.org/userslist.html> for more info.
>To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
>For additional commands, e-mail: users-help@httpd.apache.org
>
>
>
>---------------------------------------------------------------------
>The official User-To-User support forum of the Apache HTTP Server Project.
>See <URL:http://httpd.apache.org/userslist.html> for more info.
>To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
>For additional commands, e-mail: users-help@httpd.apache.org
>
>
>
--
------------------------------
Stuart Haas
IT Director
Vibe/Spin Ventures
---
Assistant Director of Technology
Miller Publishing Group
212.448.7493
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
RE: [users@httpd] Server Load Issue
Posted by Vasiliy Boulytchev <va...@boulytcheva.com>.
During the slowdown, what does /server-status/ show?
Vasiliy Boulytchev
Colorado Information Technologies, Inc.
http://www.coinfotech.com
-----Original Message-----
From: Stuart Haas [mailto:shaas@vibespin.com]
Sent: Wednesday, June 16, 2004 10:50 AM
To: users@httpd.apache.org
Subject: [users@httpd] Server Load Issue
Hi,
We are having a strange problem with Apache where the server load skyrockets
and leaves our site very unresponsive. The load average on the server
typically is well under 0.5. Once a threshold is reached (I can't pin this
down exactly, but we usually reach it several times over our busy hours),
the load immediately jumps to around 150 and more.
After a while, every thing returns to normal. I would expect that the
server load would increase linearly as traffic increases, but that is
not the case. Out site is not that busy as we serve about 30,000 pages
a day.
We are running apache 2.0.46 (Redhat's) on Redhat Enterprise Linux 3.
The hardware is 2x 3.06Ghz Xeon with 4GB RAM, so I don't believe the box is
underpowered. Our content management system is PHP based and in an effort
to solve this issue went to the Zend performance suite, but this has not had
any effect on this problem. There are no error messages in any of the logs.
How would we go about troubleshooting this issue?
Stuart
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org