MP2 + Apache2 Memory usage

[Gareth Harper <ga...@guardian.co.uk>]

We're implementing a mod_perl module for a pretty high traffic site, 
I've run some benchmarks and our mod_perl code, doing the same job as 
our current (proprietry module) is 30-40% faster.  However, the memory 
usage of mod_perl is substantially larger**. 

Having tweaked mod_perl with some preloading and other things a pmap on 
solaris shows our apache processes using 4-5Mb of private memory (mainly 
heap), the full process size is around 20Mb.

The proprietry module we currently have has process sizes of around 
16Mb, but their private data section on pmap is around 200-300k.

I know perl is somewhat more memory intensive, however I remain 
convinced that each of our modules is not using 4-5 Mb of real memory 
(7Mb before I started preloading modules), and that actually most of the 
heap is still actually "copy-on-write" and the perl processes are merely 
sharing their code on the heap, however I have no idea how to actually 
get a dump of the memory status of a process and tell which segments of 
it are still "copy-on-write". 

If anyone can give me any pointers how I might track all this info down, 
or alternatively any good ways of reducing the memory usage any further, 
I would be very grateful.  The issue arises because  we need to run up 
to 800 apache process on each of our frontend webservers at peak times 
(mainly to deal with slow connections), which would probably require (at 
1-2Gb real memory per CPU) 16-32Gb swap space.  Our sysadmin is 
understandably a little nervous about this.


One other issue I had was that I can preload all of the modules in 
startup.pl, however each module still retains the extra memory unless I 
actually remove the uses/requires from the module which is using it.  
This is fine but since we are using objects in perl this causes an error 
(unless I'm missing something obvious, which is possibly the case).

Thanks
Gareth

** Yes the obvious solution to this is to go with threaded, however 
thats the "next step" and requires some more work on making our "other" 
apache modules reentrant.  I'm pretty sure making our perl code 
threadsafe is fairly easy, however we have several other apache modules 
we've converted to apache2, but are still written in C.

-- 
Report problems: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html

Re: MP2 + Apache2 Memory usage

[Stas Bekman <st...@stason.org>]

Perrin Harkins wrote:
>>** Yes the obvious solution to this is to go with threaded
> 
> 
> Actually, reports so far are that running mod_perl 2 with threaded MPMs
> on Linux actually takes more memory because it breaks the copy-on-write
> sharing.  Preforking is more efficient.

I'd like to second Perrin. Start with the proved prefork. At some point we 
may get to figure out how to get the threaded use less memory (most likely 
when perl improves ithreads).

-- 
__________________________________________________________________
Stas Bekman            JAm_pH ------> Just Another mod_perl Hacker
http://stason.org/     mod_perl Guide ---> http://perl.apache.org
mailto:stas@stason.org http://use.perl.org http://apacheweek.com
http://modperlbook.org http://apache.org   http://ticketmaster.com

-- 
Report problems: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html

Re: MP2 + Apache2 Memory usage

[Perrin Harkins <pe...@elem.com>]

On Wed, 2004-12-15 at 17:52 +0000, Gareth Harper wrote:
> Having tweaked mod_perl with some preloading and other things a pmap on 
> solaris shows our apache processes using 4-5Mb of private memory (mainly 
> heap), the full process size is around 20Mb.

Sorry to break it to you, Gareth, but that's pretty standard.  I usually
see about 5 MB of unshared memory per process on Linux.  I check how
much is shared by looking at top, which does take into account the
copy-on-write stuff on Linux.

> If anyone can give me any pointers how I might track all this info down, 
> or alternatively any good ways of reducing the memory usage any further, 
> I would be very grateful.

Most of the documentation on reducing memory usage is here:
http://perl.apache.org/docs/1.0/guide/performance.html

> The issue arises because we need to run up 
> to 800 apache process on each of our frontend webservers at peak times 
> (mainly to deal with slow connections)

You should handle these with a front-end proxy.  You will typically only
need a fairly small number of mod_perl processes to keep 800 apache
processes busy.  The front-end proxy (or reverse proxy) approach is also
documented in that same section.

> One other issue I had was that I can preload all of the modules in 
> startup.pl, however each module still retains the extra memory unless I 
> actually remove the uses/requires from the module which is using it.

That means something is wrong with your pre-loading.  It should not make
any difference at all if you have use/require stuff in your modules
after pre-loading them.

> ** Yes the obvious solution to this is to go with threaded

Actually, reports so far are that running mod_perl 2 with threaded MPMs
on Linux actually takes more memory because it breaks the copy-on-write
sharing.  Preforking is more efficient.

- Perrin


-- 
Report problems: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html