Swapping

[Mike Norton <mi...@jobsite.co.uk>]

We are currently experiencing some problems with our mod_perl setup the server we are using is a Dual Xeon 2.4Ghz running Redhat AS 2.1, 4GB Ram , Apache/1.3.27 (Unix) mod_perl/1.27. When the server is under a heavy load it all of a sudden consumes a large amount of swap and the server load shoots through the roof, the machine normally sits quite happily not even touching the swap normally however it seems to consume anything up to 3GB of swap all in one go. Does anyone have any ideas on how to resolve this issue or why it happens ?
 
Thanks
 
Mike

Re: Swapping

[Perrin Harkins <pe...@elem.com>]

On Mon, 2003-10-13 at 11:02, Kyle Dawkins wrote:
> You must, at ALL times, keep your machine from swapping out Apache
> children.

Good advice.  To go a step firther, this is all covered in detail in the
mod_perl documentation here:
http://perl.apache.org/docs/1.0/guide/performance.html#Performance_Tuning_by_Tweaking_Apache_Configuration

In addition, I would suggest using a much higher MaxRequestsPerChild
setting, but using something like Apache::SizeLimit to keep the size of
the processes from growing indefinitely.

- Perrin

Re: Swapping

[Kyle Dawkins <ky...@centralparksoftware.com>]

Mike

I'm pretty sure you're experiencing a standard problem that most of us 
have been bitten by at one stage or another.  Ged is right when he says 
that 120 apache children at 40MB a pop will eat your RAM up.  The piece 
of the puzzle that you're missing is this:  due to the dopey VM code in 
Linux, when you run out of physical ram, processes get swapped out and 
at that instant, they become UNSHARED.  In other words, if your process 
is taking up 40MB, of which 30MB is shared, and it gets swapped out, it 
will get swapped out as a 40MB process.  When it gets swapped back in, 
it will take up 40MB of your precious RAM, and will probably force 
another apache process to get swapped out and unshared, which will 
cause another...

It's a tricky problem, but here's how you solve it:

You must, at ALL times, keep your machine from swapping out Apache 
children.  This is tricky.  If you're running other things on your 
machine, like a DB (I assume you're not since you say you're using 
Oracle) then it becomes nearly impossible to prevent.  But if your 
machine is pretty much just Linux + Apache, you're OK: just make sure 
that

MaxClients * LargestPossibleChildSize + AllRamUsedByOSAndOtherProcesses 
< 4GB

Since you're running a front-end Apache too, you need to figure that 
into the equation.  But the key here is that you can NEVER swap out an 
apache process, or you'll end up in the downward spiral of death, which 
seems to be what's happening to you.  There's no recovery from that 
unless you can stop apache; only other solution is to reboot.  If 
that's the case, you might as well be using Windows XP.

Hope this helps

Kyle Dawkins
Central Park Software


On Monday, Oct 13, 2003, at 05:34 US/Pacific, Mike Norton wrote:

> We are currently experiencing some problems with our mod_perl setup 
> the server we are using is a Dual Xeon 2.4Ghz running Redhat AS 2.1, 
> 4GB Ram , Apache/1.3.27 (Unix) mod_perl/1.27. When the server is 
> under a heavy load it all of a sudden consumes a large amount of swap 
> and the server load shoots through the roof, the machine normally sits 
> quite happily not even touching the swap normally however it seems to 
> consume anything up to 3GB of swap all in one go. Does anyone have any 
> ideas on how to resolve this issue or why it happens ?
>  
> Thanks
>  
> Mike

Re: Swapping

[Ged Haywood <ge...@www2.jubileegroup.co.uk>]

Hi there,

On Mon, 13 Oct 2003, Mike Norton wrote:

> [snip] under a heavy load it all of a sudden consumes a large amount
> of swap and the server load shoots through the roof [snip] Does
> anyone have any ideas on how to resolve this issue or why it happens

No idea why it happens, you haven't given anything like enough
information, but it might be that your server is spawning many
processes because the processes that are in use are all busy.
This sort of thing has been discussed a lot on this List, but
not much recently as I recall.

There are Apache configuration variables and tools for limiting the
number of processes (particularly mod_perl processes) which you might
want to investigate.  The performance tuning section in the Guide
may have some ideas for you.

There's a possibility that Apache children are using up memory they
shouldn't.  If that's a suspicion then have a look at the setting of
MaxRequestsPerChild.  If it's 0 then I'd suggest setting it to
something in the hundreds to see if that helps.

If we are to be able to give more specific help you need to do some
investigation and let us have more information about your server, its
configuration, what it's doing, the processes that you see, etc.

73,
Ged.