Re: Apache/BSDI bug

[Chuck Murcko <ch...@telebase.com>]

Forwarded message from Dave Yadallee:
> 
> > From: Chuck Murcko <ch...@telebase.com>
> >
> > Chris Ochs <ch...@valleynet.net> wrote:
> > >
> > > We have been running Apache for about a year now, and even with the latest
> > > beta 1.1b2 version under heavy loads the server hangs.  It usually happens
> > > when the server has around 120-150 child processes.
> > >
> > > What happens is that the server will pretty much just freeze up and/or slow
> > > to a crawl.
> > >
> > > We have tried every trick we can think of to fix it, but no luck.  max proc
> > > and FD limits are both 256 and SOMAXCON in the kernel is set to 64.
> > >
> >
> > If you haven't run out of mbufs or memory, you might need to raise
> > CHILD_MAX; the default for 2.x is 64. Vmstat(8) will tell you about
> > memory problems, and netstat -m will tell you if you've got enough
> > mbufs.
> >
> > chuck
>  
> This is a bug that should be reported to Apache.
>  
> I am running the Apache beta sans problem, but Apache needs to know
> of any potential bugs.
>  
I have in fact cc:ed this msg. to the Apache developer's list, since I am
one of them.

I had exactly this sort of problem last December with a 1.0.x version of
Apache and found out that

1) My syslog buffers were getting stuffed with messages from a logging
CGI script, indirectly causing network sluggishness on the Apache machine.

2) The gateway that said script forwarded through had insufficient memory,
and was becoming unresponsive network-wise.

3) A Sparc-20 beyond the gateway (the ultimate destination for the CGI
forwarded data) was also swamped, causing further backup of connections to
the front-wnd (Apache) machine, which by now was peaking at > 170 children.
Note: I had MaxClients cranked to 200; see below.

I used vmstat and netstat -m to check out things on the Apache and gateway
boxen. The Sparc was not under my control, but I made some suggestions there.

The result:

16 more MB of RAM in both the Apache (48 MB) and gateway (32 MB) boxes,
and the following config file values:

maxusers        256

# Network options.  NMBCLUSTERS defines the number of mbufs and defaults to
# 256.  This machine is a server that handles lots of traffic, so we crank
# that value.
options         SOMAXCONN=256           # max pending connects
options         NMBCLUSTERS=4096        # mbufs at 4096

#
# Misc. options
#
options         CHILD_MAX=512           # maximum number of child processes
options         OPEN_MAX=512            # maximum fds (breaks RPC svcs)

These are the values that changed from the stock GENERIC ones.

Don't try that many NMBCLUSTERS unless you have at least 32 MB of RAM
in your machine; it will fail to boot otherwise (at least under 2.0.x
BSDI).

More than 256 fds breaks RPC services under 2.0.x (haven't tried under
2.1 yet). This is a hardcoded value in the RPC library. I don't run
RPC on these machines.

Now these machines can run at the 150 child value and beyond without
getting sluggish.

>From the Apache httpd.conf:

# Limit on total number of servers running, i.e., limit on the number
# of clients who can simultaneously connect --- if this limit is ever
# reached, clients will be LOCKED OUT, so it should NOT BE SET TOO LOW.
# It is intended mainly as a brake to keep a runaway server from taking
# Unix with it as it spirals down...

MaxClients 150

I'd like to try and find out if this machine is being beaten to death
before considering this an Apache bug, since no other users have reported
such problems, and a number of them are running systems that sound as
busy as Chris'.

One other question for Chris: Are you getting any system or Apache messages
in your logs that are peculiar to your machine being sluggish?

chuck
Chuck Murcko	N2K Inc.	Wayne PA	chuck@telebase.com
And now, on a lighter note:
"I must have a prodigious quantity of mind; it takes me as much as a
week sometimes to make it up."
		-- Mark Twain, "The Innocents Abroad"

Re: Apache/BSDI bug

[Brian Behlendorf <br...@organic.com>]

Chuck (and the rest of you) - feel free to add any comments you might 
have to http://www.apache.org/docs/perf.html.  Last night Jeff Mogul from 
DEC gave me two newsgroup postings which he made describing how to tweak 
kernel parameters in Digital Unix... it was extensive enough that it 
merited its own page, if you want to buff out the BSDI section even more 
that could be made into its own page as well.  Of course, it's far better 
to be able to point to a (hopefully updated) page on performance tuning 
hosted by the OS vendors themselves....

	Brian

--=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=--
brian@organic.com  |  We're hiring!  http://www.organic.com/Home/Info/Jobs/