You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@httpd.apache.org by greg boyington <ev...@gmail.com> on 2007/11/15 18:20:44 UTC
[users@httpd] debugging strategies for httpd that kills the box
Hello List,
I have a FreeBSD 5.5 (amd64) server with the following:
Apache/2.0.61 (FreeBSD)
mod_ssl/2.0.53
OpenSSL/0.9.7e-p1
PHP/4.4.4 with Suhosin-Patch
mod_apreq2-20051231/2.6.0
mod_perl/2.0.3
Perl/v5.8.6
Background: The server was recently updated to FreeBSD 5.5 from 5.3,
which had run for a couple of years without interruption. I'm not
especially convinced the issue I'm having is related to this upgrade,
but it's the most significant change in some time, so it's worth
mentioning. Apache is configured with KeepAlive on, as this box
provides a number of other services, but I noticed last week that the
majority of the requests (specifically those being handled by
HTML::Mason via mod_perl) were not sending a Content-Length header,
and thus the KeepAlive settings were largely useless. I added this
header to the mod_perl output, and began tuning apache to maximize
performance. I got, for us, quite acceptable results using the
following:
<IfModule prefork.c>
StartServers 5
MinSpareServers 5
MaxSpareServers 10
MaxClients 200
MaxRequestsPerChild 0
</IfModule>
The problem: After a few days the server became unstable. Twice the
server was out of swap space and had to be rebooted. While attempting
to solve this problem, I came to understand that the webserver seemed
responsible; for as-yet unknown reason, the box will suddenly lock up,
and if I do not kill httpd within 5 or 10 seconds the server will
become completely unresponsive and need to be rebooted. If left
alone, swap space will be consumed and the kernel will begin killing
off httpd processes, but not fast enough. By altering the apache
config to the following, I am able to avoid the hang:
<IfModule prefork.c>
StartServers 5
MinSpareServers 5
MaxSpareServers 5
MaxClients 40
MaxRequestsPerChild 0
</IfModule>
But naturally this provides less than optimal performance. Other
things I tried, without impact:
-- setting KeepAlive off
-- disabling all vhosts
-- disabling mod_perl2
-- disabling mod_php
-- running httpd -X
-- tweaking/disabling resource limits via Apache2::Resource
-- recompiling httpd
-- recompiling mod_perl2
-- recompiling & reinstalling the freebsd userland
The server has 1gb of ram, 4gb of swap (I recently doubled it in an
attempt to give me more time to diagnose the problem when it occurs --
didn't help, as the lock-up occurs long before swap is consumed), and
in the moments before the hang, shows no particular level of activity
-- swap usage is around 100mb, load is very low (1.5 - 2.0 on a
dual-proc system). There doesn't seem to be any correlation to
specific requests or request types, whether static, cgi, mod_perl or
php. There are no errors in the server logs prior to the lock up (
although once, and only once, i saw "httpd in free(): error: page is
already free" ).
I'm hoping someone here can make some suggestions on how to
troubleshoot the problem further, because I'm kind of at a loss.
I have not (yet) posted this to the mod_perl list or freebsd-users,
though I would be happy to do so if it's thought the problem might lie
within those realms.
Thanks!
Greg
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
[users@httpd] compile apache 2.2.0 for IPv6 address.
Posted by Ashwani Kumar Sharma <As...@mindtree.com>.
Hi All,
I compiled apache 2.2.0 successfully on Windows.
But apache is not responding to IPv6 address. It says "Internet explorer can
not display the page"
What changes I should do in the configuration file.
Do I need to compile the source code with some IPv6 option flag or something?
Please reply.
PS: I have configured my machine for IPv6 address. I am able to ping
successfully.
Thanks and Regards,
Ashwani Sharma
Mob: +91+9916454843
Off: +91-80-26265053
DISCLAIMER:
This message (including attachment if any) is confidential and may be privileged. If you have received this message by mistake please notify the sender by return e-mail and delete this message from your system. Any unauthorized use or dissemination of this message in whole or in part is strictly prohibited.
E-mail may contain viruses. Before opening attachments please check them for viruses and defects. While MindTree Consulting Limited (MindTree) has put in place checks to minimize the risks, MindTree will not be responsible for any viruses or defects or any forwarded attachments emanating either from within MindTree or outside.
Please note that e-mails are susceptible to change and MindTree shall not be liable for any improper, untimely or incomplete transmission.
MindTree reserves the right to monitor and review the content of all messages sent to or from MindTree e-mail address. Messages sent to or from this e-mail address may be stored on the MindTree e-mail system or else where.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] debugging strategies for httpd that kills the box
Posted by Greg Boyington <gr...@regex.ca>.
On Nov 16, 2007 10:39 AM, Christian Folini <ch...@post.ch> wrote:
> On Fri, Nov 16, 2007 at 10:06:50AM -0500, Greg Boyington wrote:
> > > - Play around with MaxRequestsPerChild. A typical value
> >
> > I will do -- I experimented with low values (100-1000) in addition to
> > 0, but nothing seemed to correlate. In my testing, the hangs would
> > often occur long before this threshold was ever reached.
>
> The directive means to let a child process die and fork
> it anew after n individual request for this particular
> child process.
Indeed; I monitored the request counts via Apache2::VMonitor;
sometimes processes would reach the MaxRequestsPerChild and die/fork
before the lockup, sometimes not.
I will experiment with mod_status and mod_diagnostics once the box is
off the wire; thanks for the suggestions. I'll post my progress, if
there is any. :)
-G
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] debugging strategies for httpd that kills the box
Posted by Christian Folini <ch...@post.ch>.
On Fri, Nov 16, 2007 at 10:06:50AM -0500, Greg Boyington wrote:
> > - Play around with MaxRequestsPerChild. A typical value
>
> I will do -- I experimented with low values (100-1000) in addition to
> 0, but nothing seemed to correlate. In my testing, the hangs would
> often occur long before this threshold was ever reached.
The directive means to let a child process die and fork
it anew after n individual request for this particular
child process.
You can try and keep an eye on these children with mod_status,
write it all into a logfile and reconstruct when the lockup
occurs. Not sure it's of any use though. ;(
You mentioned httpd -X. Maybe you try and follow
the process(es) during the lockup with strace/truss.
Besides, I forgot to mention, that there is mod_diagnostics
by Nick Kew. Itis mentioned here from time to time.
See http://apache.webthing.com/mod_diagnostics/
Nick will be able to give you specific hints on how to use
it.
mod_forensic can also help to nail down the bad request. But it
sounds, like your problem is not about a specific request.
Otherwise, I am slowly running out of ideas and you have
done all your homework before. It's really hard to recommend
anything here. At least for me, but that does not say much.
regs,
Christian
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] debugging strategies for httpd that kills the box
Posted by Greg Boyington <gr...@regex.ca>.
On Nov 16, 2007 2:19 AM, Christian Folini <ch...@post.ch> wrote:
> Hey Greg,
Hi Christian, thanks for your response.
> - Play around with MaxRequestsPerChild. A typical value
I will do -- I experimented with low values (100-1000) in addition to
0, but nothing seemed to correlate. In my testing, the hangs would
often occur long before this threshold was ever reached.
> - Did you compile apache yourself? Give it a try.
I do. Most everything is installed from ports; I recompiled by hand
when testing.
> - Have you tried to get the lock-up in a lab setup with
> apache bench or some other load generating tool?
Yup -- ab was used to test each random tweak I made to try and isolate
the problem.
> - You disabled mod_perl2 and mod_php. This sounds as if
> you would not really need them. (Or you sacrificed
> functionality :)
They are both critical to our operation, but since the box was dead in
the water anyway I took the opportunity to turn everything off and
test each one in turn. I only mentioned them in particular because I
figured they were the most likely suspects. :)
> then you might give it a try with another mpm.
That crossed my mind as well, but I haven't had any experience with
mod_perl2 and HTML::Mason under mpm, so I would have to do some
research to see what I was getting into there. In the mean time, The
Powers That Be have decided to build a replacement box so I can pull
this one off the wire and do more testing. Hard to debug a problem
that kills the production machine resonsible for web, storefront,
email and support chat.
> Good luck - and keep us informed on how it goes.
Thanks, I will.
-G
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] debugging strategies for httpd that kills the box
Posted by Christian Folini <ch...@post.ch>.
Hey Greg,
On Thu, Nov 15, 2007 at 12:20:44PM -0500, greg boyington wrote:
> Background: The server was recently updated to FreeBSD 5.5 from 5.3,
> which had run for a couple of years without interruption. I'm not
> especially convinced the issue I'm having is related to this upgrade,
> ...
> <IfModule prefork.c>
> StartServers 5
> MinSpareServers 5
> MaxSpareServers 10
> MaxClients 200
> MaxRequestsPerChild 0
> </IfModule>
Problems as this are hard to track down. I'll give you a few thoughts
what you could do next. Maybe it helps, but I can not guarantee it.
- Play around with MaxRequestsPerChild. A typical value
outside of 0 is 10000. I guess that's also the default,
but not in your setup.
- Did you compile apache yourself? Give it a try.
- Have you tried to get the lock-up in a lab setup with
apache bench or some other load generating tool?
Guess, this would help.
- You disabled mod_perl2 and mod_php. This sounds as if
you would not really need them. (Or you sacrificed
functionality :)
Disable all modules which you do not really need. Definitely
all of them. If you can boil it down to threadsafe modules,
then you might give it a try with another mpm.
http://httpd.apache.org/docs/2.0/mpm.html
Good luck - and keep us informed on how it goes.
Christian
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org