You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@httpd.apache.org by Jacob Coby <jc...@listingbook.com> on 2002/08/19 23:47:15 UTC
Apache 1.3.26 DOS'ing itself?
Recenty Apache seems to be DOS'ing itself every couple of days. httpd
starts using 100% cpu and 100% mem. The computer starts HDD thrashing, and
eventually the server stops responding.
A check of the error_log reveals:
--- snippet ---
[Mon Aug 19 16:44:10 2002] [error] server reached MaxClients setting,
consider raising the MaxClients setting
[Mon Aug 19 16:53:55 2002] [notice] child pid 19538 exit signal Segmentation
fault (11)
[Mon Aug 19 16:55:14 2002] [warn] child process 19961 still did not exit,
sending a SIGTERM
[Mon Aug 19 16:55:22 2002] [warn] child process 19137 still did not exit,
sending a SIGTERM
--- end snippet ---
Which later becomes:
--- snippet ---
[Mon Aug 19 16:55:26 2002] [error] child process 19961 still did not exit,
sending a SIGKILL
[Mon Aug 19 16:55:26 2002] [error] child process 19137 still did not exit,
sending a SIGKILL
[Mon Aug 19 16:55:26 2002] [error] child process 19667 still did not exit,
sending a SIGKILL
--- end snippet ---
I'm assuming this has something to do with MaxClients? A search of google
[groups] shows some other cases of this happening, but no resolutions.
My server settings are as follows:
KeepAliveTimeout 15
MinSpareServers 10
MaxSpareServers 20
StartServers 20
MaxClients 100
MaxRequestsPerChild 100
I'm guessing that when the server hits MaxClients, all hell breaks loose for
some reason? What do I do? I'd really rather not decrease maxclients
drastically unless absolutly necessary. I've lowered it to 90 for the time
being.
Any help is much appreciated. We've just weaned some large companies from
proprietary systems to our system, and it's not very good to have these
random timeouts.
-Jacob
http://www.listingbook.com
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: Apache 1.3.26 DOS'ing itself still
Posted by Jacob Coby <jc...@listingbook.com>.
> That's possible. You should NEVER make your web server use swap space.
> MaxClients should be set low enough such that all serving happens from
> main memory.
>
> It sounds like your httpd processes are using quite a bit of memory.
> You might want to cut down on unnecessary modules.
The more I think about it, the more plausable this situation is. I forgot
to mention that our website is written in PHP, and it has a mem limit of
24mb. I'm pretty sure that its a problem of physically running out of
memory. All it takes is two people running a large query, and suddenly that
40mb free is gone. And then some.
Looks like another 256mb is going to need to be ordered.
> Also see:
> http://httpd.apache.org/docs/misc/perf-tuning.html
> for a few important tips.
Thanks for the info, I'm reading that now.
-Jacob
http://www.listingbook.com
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: Apache 1.3.26 DOS'ing itself still
Posted by Joshua Slive <jo...@slive.ca>.
Jacob Coby wrote:
> I almost wonder if i'm exhausting the physical memory, and its causing
> excessive swapping. This server only has 256mb of ram. With 35 instances
> of httpd running, I have 40mb free. I'm guessing that an additonal 50 may
> take up more than that 40mb?
That's possible. You should NEVER make your web server use swap space.
MaxClients should be set low enough such that all serving happens from
main memory.
It sounds like your httpd processes are using quite a bit of memory.
You might want to cut down on unnecessary modules.
Also see:
http://httpd.apache.org/docs/misc/perf-tuning.html
for a few important tips.
Turning down KeepAliveTimeOut may help a little, but it is best not to
turn it off if you can avoid it.
Joshua.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Apache 1.3.26 DOS'ing itself still
Posted by Jacob Coby <jc...@listingbook.com>.
Apache ran away again, happily bringing down the webserver. This time,
however, it didn't reach the MaxClients setting before it started grinding
on the disks.
Between the last instance and this one, I raised maxclients to 125 (from
100) and set MaxRequestsPerChild to 100000 (from 100).
After this latest panic, I've set MaxRequestPerChild to 0.
What can I do to diagnose this problem? Will disabling KeepAlive help?
What will it do to performance?
I almost wonder if i'm exhausting the physical memory, and its causing
excessive swapping. This server only has 256mb of ram. With 35 instances
of httpd running, I have 40mb free. I'm guessing that an additonal 50 may
take up more than that 40mb?
Thanks,
-Jacob
http://www.listingbook.com
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: Apache 1.3.26 DOS'ing itself?
Posted by Jacob Coby <jc...@listingbook.com>.
> Well, this may not be the "offocial" answer, but:
>
> 1) you need to raise the maxclients setting, not lower it. Your server
> usage is going up, and it'll take more apache processes going to handle
> that. Note that if you simply don't have the hardware resources to handle
> the requests that's a different matter :-)
Excellent, thank you! I've raised it to 120, and will continue raising it
until I exhaust the memory. With 120 max, I've still got 40mb free. Not
sure how much further I can go without impacting server performance due to
swapping. We're getting a second storage server/loadbalancing server in the
next week or so, so I'm hoping to be able to handle ~400 simul connections
between the two comps. More than enough to saturate the T1 line :)
> 2) The 100% mem/100% cpu thing is most likely due to your *extremely*
> low MaxRequestsPerChild setting. You have 100 servers running, and those
> servers will *exit* and *fork* every 100 requests they get. That means
> that you're probably spending as much time in fork() and wait() as you
> are actually serving requests. At the very least, raise that setting
> a lot; if you can, change it to 0. This works on my linux box for 1.3,
> at least. If you can't make it 0, try something like 10000 or 100000.
I see. That was a carryover from the last sysadmin's setup. I don't know
why it was set so low, except to perhaps prevent PHP mem leaks? I've raised
it to 10000 for the time being. I'll raise it in 10k increments until it
hits 100000 or something goes bad. Once I reach 100000, I'll move it to 0.
-Jacob
http://www.listingbook.com
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: Apache 1.3.26 DOS'ing itself?
Posted by Michael <mi...@asstr.org>.
Well, this may not be the "offocial" answer, but:
1) you need to raise the maxclients setting, not lower it. Your server
usage is going up, and it'll take more apache processes going to handle
that. Note that if you simply don't have the hardware resources to handle
the requests that's a different matter :-)
In general, though, unless you're afraid of apache overwhelming your
server and taking it down then you should be safe to raise maxclients
up until you reach the hard process limit for your server (you'll have to
recompile apache to raise it above 256).
2) The 100% mem/100% cpu thing is most likely due to your *extremely*
low MaxRequestsPerChild setting. You have 100 servers running, and those
servers will *exit* and *fork* every 100 requests they get. That means
that you're probably spending as much time in fork() and wait() as you
are actually serving requests. At the very least, raise that setting
a lot; if you can, change it to 0. This works on my linux box for 1.3,
at least. If you can't make it 0, try something like 10000 or 100000.
- Michael
On Mon, 19 Aug 2002, Jacob Coby wrote:
> Recenty Apache seems to be DOS'ing itself every couple of days. httpd
> starts using 100% cpu and 100% mem. The computer starts HDD thrashing, and
> eventually the server stops responding.
>
> A check of the error_log reveals:
>
> --- snippet ---
> [Mon Aug 19 16:44:10 2002] [error] server reached MaxClients setting,
> consider raising the MaxClients setting
> [Mon Aug 19 16:53:55 2002] [notice] child pid 19538 exit signal Segmentation
> fault (11)
> [Mon Aug 19 16:55:14 2002] [warn] child process 19961 still did not exit,
> sending a SIGTERM
> [Mon Aug 19 16:55:22 2002] [warn] child process 19137 still did not exit,
> sending a SIGTERM
> --- end snippet ---
>
> Which later becomes:
>
> --- snippet ---
> [Mon Aug 19 16:55:26 2002] [error] child process 19961 still did not exit,
> sending a SIGKILL
> [Mon Aug 19 16:55:26 2002] [error] child process 19137 still did not exit,
> sending a SIGKILL
> [Mon Aug 19 16:55:26 2002] [error] child process 19667 still did not exit,
> sending a SIGKILL
> --- end snippet ---
>
> I'm assuming this has something to do with MaxClients? A search of google
> [groups] shows some other cases of this happening, but no resolutions.
>
> My server settings are as follows:
>
> KeepAliveTimeout 15
> MinSpareServers 10
> MaxSpareServers 20
> StartServers 20
> MaxClients 100
> MaxRequestsPerChild 100
>
> I'm guessing that when the server hits MaxClients, all hell breaks loose for
> some reason? What do I do? I'd really rather not decrease maxclients
> drastically unless absolutly necessary. I've lowered it to 90 for the time
> being.
>
> Any help is much appreciated. We've just weaned some large companies from
> proprietary systems to our system, and it's not very good to have these
> random timeouts.
>
> -Jacob
> http://www.listingbook.com
>
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
RE: Apache 1.3.26 DOS'ing itself?
Posted by Koen Vingerhoets <ko...@ubench.com>.
Hi,
I have it at 256, which is the max...and I get the error too...
Error_log said: recompile apache to get over the 256 limit :(
Koen
Met vriendelijke groet,
Koen Vingerhoets
***** UBench nv *****
http://www.ubench.com
____________________________________________
The information contained in this electronic mail message is privileged and
confidential,
and is intended only for use of the addressee. If you are not the intended
recipient, you
are hereby notified that any disclosure,reproduction, distribution or other
use of this
communication is strictly prohibited.
If you have received this communication in error, please notify the sender
by reply
transmission and delete the message without copying or disclosing it.
-----Original Message-----
From: Jacob Coby [mailto:jcoby@listingbook.com]
Sent: 19 August 2002 23:47
To: users@httpd.apache.org
Subject: Apache 1.3.26 DOS'ing itself?
Recenty Apache seems to be DOS'ing itself every couple of days. httpd
starts using 100% cpu and 100% mem. The computer starts HDD thrashing, and
eventually the server stops responding.
A check of the error_log reveals:
--- snippet ---
[Mon Aug 19 16:44:10 2002] [error] server reached MaxClients setting,
consider raising the MaxClients setting
[Mon Aug 19 16:53:55 2002] [notice] child pid 19538 exit signal Segmentation
fault (11)
[Mon Aug 19 16:55:14 2002] [warn] child process 19961 still did not exit,
sending a SIGTERM
[Mon Aug 19 16:55:22 2002] [warn] child process 19137 still did not exit,
sending a SIGTERM
--- end snippet ---
Which later becomes:
--- snippet ---
[Mon Aug 19 16:55:26 2002] [error] child process 19961 still did not exit,
sending a SIGKILL
[Mon Aug 19 16:55:26 2002] [error] child process 19137 still did not exit,
sending a SIGKILL
[Mon Aug 19 16:55:26 2002] [error] child process 19667 still did not exit,
sending a SIGKILL
--- end snippet ---
I'm assuming this has something to do with MaxClients? A search of google
[groups] shows some other cases of this happening, but no resolutions.
My server settings are as follows:
KeepAliveTimeout 15
MinSpareServers 10
MaxSpareServers 20
StartServers 20
MaxClients 100
MaxRequestsPerChild 100
I'm guessing that when the server hits MaxClients, all hell breaks loose for
some reason? What do I do? I'd really rather not decrease maxclients
drastically unless absolutly necessary. I've lowered it to 90 for the time
being.
Any help is much appreciated. We've just weaned some large companies from
proprietary systems to our system, and it's not very good to have these
random timeouts.
-Jacob
http://www.listingbook.com
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org