You are viewing a plain text version of this content. The canonical link for it is here.
Posted to modperl@perl.apache.org by Kjetil Kjernsmo <kj...@opera.com> on 2006/11/06 17:48:05 UTC
Loading at startup to use shared memory
Hi all!
I'm working on improving the scalability and performance of our app. It
is rather bad, and I noticed that not many minutes after restart, the
amount of shared memory is down to 10% of the total memory (and I'm at
least at 1/3 on my home Axkit::App::TABOO), and besides, it is bad from
the start too.
Obviously, there is a lot to gain here, since we run a lot of children
across many backends.
So, I pulled out Stas and Eric's book, which has a chapter about it.
That's about mod_perl 1.0, but much of it I presume still is valid.
Also, I found an example in
http://perl.apache.org/docs/2.0/user/handlers/server.html#Startup_File
which I tried.
I have not used the example script of Stas and Eric, but instead insert
some GTop stuff in a registry script that loads a small picture from
the database. It doesn't really use a lot of the stuff I load at
startup. Then I warn the numbers, to find them in the Apache log.
Then, I have stared at the numbers with bewilderment.
Without any startup.pl file:
Size: 59346944 Shared: 6172672 Unshared: 53174272
Then I set the startup.pl, but removed and included different modules,
just to see what happens. The results are consistent, though:
Size: 76079104 Shared: 5763072 Unshared: 70316032
Size: 78835712 Shared: 5947392 Unshared: 72888320
Size: 73920512 Shared: 4878336 Unshared: 69042176
So, what I see is that consistently, the amount of shared memory goes
down when I preload modules at server startup... Weird, huh?
It is pretty clear that total memory per child must go up by preloading
modules, but the idea was that the shared portion of it should be much
greater... I mean, just the code, which is identical per child, should
contribute to this...
I've been monitoring top as the server runs too, and it seems consistent
with these findings, also on other parts of the server. I guess we
should rewrite a lot of stuff here, but I was hoping this would be a
quick fix for some of our problems, as I think there is a lot in here
that would reasonably be in shared memory pages. So, any ideas where I
can start to understand why this isn't shared?
Cheers,
Kjetil
--
Kjetil Kjernsmo
Information Systems Developer
Opera Software ASA
Re: Loading at startup to use shared memory
Posted by Jonathan Vanasco <mo...@2xlp.com>.
On Nov 7, 2006, at 1:03 PM, Perrin Harkins wrote:
> I think some of these are a little over-zealous, Jonathan.
oh, they're completely off the hook. but they work.
BTW, none of my approaches were pre mature optimization. They were
all based on profiling code via Apache::Status to get better
performance.
> There are always trade-offs when using CPAN modules, but when in
> doubt,
> it's a good idea to use them first and then make changes later after
> your code is working, if your tests show that one of them is a memory
> hog. File system traversal always sounds easier than it really is, as
> is often the case with commonly used CPAN modules.
FileFind, like many cpan modules, is good if you're doing some
awkward tasks on a wide variety of platforms. If i made a
distributed product or library, i'd use it.
But some 'simple' tasks on a known / unchanging configuration are
sometimes best kept simple. If i want to get all of the .html files
in a path, and all my servers are freebsd, and I only install this
app on my servers -- file system traversal is really simple. if i
had to support linux or win, then i'd have issues. stripping out
file::find saved me 2mb of memory.
> Your debug statements take up 40% of your memory? I don't think that
> will be the case for most people.
My own code was about 20. On average a CPAN module was around 40. I
tried doing some regex on RoseDB and Petal, which are my biggest
libraries. Some modules in each went down 2% (nothing). But other
modules went down 60% in size. The difference was between importing
all of the random debugging modules and functions into namespace, and
just flat-out omitting a lot of logic and strings.
> I hardly ever use perl's pseudo-constants. They're more trouble
> than they're worth, and most
> apps will not show a significant difference in performance. This
> is only worth thinking about if you have huge amounts of debugging
> statements in your code (in which case, maybe you should try using
> the debugger instead).
or if you have 'intensive' debugging statements in code.
I definitely use them too much myself. I haven't seen any real
difference in performance. the only difference i've seen is in
memory use. a little over zealousness, and i got another MP2 server
AND 50mb more memory for Postgres (which is sadly running on the same
box right now, the reason why i had to go crazy ).
// Jonathan Vanasco
| - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - - - - - - - - - - - - - -
| FindMeOn.com - The cure for Multiple Web Personality Disorder
| Web Identity Management and 3D Social Networking
| - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - - - - - - - - - - - - - -
| RoadSound.com - Tools For Bands, Stuff For Fans
| Collaborative Online Management And Syndication Tools
| - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - - - - - - - - - - - - - -
Re: Loading at startup to use shared memory
Posted by Perrin Harkins <pe...@elem.com>.
On Tue, 2006-11-07 at 11:35 -0500, Jonathan Vanasco wrote:
> some tips:
I think some of these are a little over-zealous, Jonathan.
> but things like File::Find eat up a TON of memory ( 2+ mb ), and
> most functionality can be coded in 10-15 minutes. ( vs 2 minutes if
> you use the module )
There are always trade-offs when using CPAN modules, but when in doubt,
it's a good idea to use them first and then make changes later after
your code is working, if your tests show that one of them is a memory
hog. File system traversal always sounds easier than it really is, as
is often the case with commonly used CPAN modules.
> use all of your internal code with constants for debugging, not
> variables
> if ( DEBUG__ ) { } is optimized away
> if ( $DEBUG__ ) {} is not
> if you want to use variables, wrap them in a constant
> if ( DEBUG ) {
> if ( $DEBUG__ ) {}
> }
> your dev machine takes a hit on performance with this approach, but
> your production machine is way leaner ( 30-40% less opcodes and
> memory ) than the standard $DEBUG methods
Your debug statements take up 40% of your memory? I don't think that
will be the case for most people. I hardly ever use perl's
pseudo-constants. They're more trouble than they're worth, and most
apps will not show a significant difference in performance. This is
only worth thinking about if you have huge amounts of debugging
statements in your code (in which case, maybe you should try using the
debugger instead).
> Note that DBI and some other modules do caching.
> DBI caches db handles and statement handles internally.
Usually you tell it to do this for performance reasons, by using
Apache::DBI or DBI->connect_cached and prepare_cached.
- Perrin
Re: Loading at startup to use shared memory
Posted by Jonathan Vanasco <mo...@2xlp.com>.
On Nov 7, 2006, at 6:57 AM, Kjetil Kjernsmo wrote:
> Exactly what numbers are you then reading out of top then?
Run your app on a clean box -- ie: don't run anything other than
apache/mp ( if you can toss the db onto another service , do it ).
but turn off everything else.
Reboot your machine into that state, so all the memory is
'free' ( nothing is marked as used when its free, and then recycled )
then watch the 'free' system memory as you start / stop apache. if
all goes well, you'll have no memory 'leakage' - it'll stop to the
same amount of free ram as before. do a few test runs ( its not
really a leak, the memory is free , but the system sometimes doesn't
show it as free ). profile your code, if needed, to get things to
show like that.
then just calculate differences on each round of server configurations.
also:
using the various apache sizing tools like Apache2::Status with all
the bells and whistles, you can profile where code /memory is wasted
some tips:
avoid cpan modules not intended for persistent frameworks if you can.
when you use cpan stuff, opt for ones that wrap c functions.
just weigh the benefit from the drawbacks.
things like DBI , Rose/DBIx, Date::Calc, etc are obviously integral.
but things like File::Find eat up a TON of memory ( 2+ mb ), and
most functionality can be coded in 10-15 minutes. ( vs 2 minutes if
you use the module )
use all of your internal code with constants for debugging, not
variables
if ( DEBUG__ ) { } is optimized away
if ( $DEBUG__ ) {} is not
if you want to use variables, wrap them in a constant
if ( DEBUG ) {
if ( $DEBUG__ ) {}
}
your dev machine takes a hit on performance with this approach, but
your production machine is way leaner ( 30-40% less opcodes and
memory ) than the standard $DEBUG methods
use refs whenever you can.
i have a custom page renderer. it doesn't print directly to $r,
because it builds page components ( usually out of Petal but
sometimes other stuff too ) then aggregates them together.
switching to a ref based system saved me ~3mb of memory per child
on the render section alone. that was about 5 lines of code to
implement.
pay close attention to what is loaded pre-fork and unshared post-fork
if you hit a certain page, and your unshared memory in that child
jumps up 10+mb, you should look into exactly what is causing that
jump. you probably have something that can be optimized
Find your 'baseline' apache memory imprints
Run single server mode. Note the size of memory before any page
hits. Then note growth as you hit the system.
Run multi server mode. Note the size of the parent process, the
child process BEFORE any hits. Then note the size of both after a hit.
In my system,
I'm up to ~110 parent memory.
Immediately after the fork, each child is ~5mb.
Immediately after the first request, each child is ~12mb.
subsequent requests grow a few k / mb depending on output. i kill
them @500 requests per child or 20MB, whichever comes first (under
normal use, its the same )
Note that DBI and some other modules do caching.
DBI caches db handles and statement handles internally. i think
they reset @ 120.
usually that means you see 1) process growth from old handles
sticking around ; 2) leaked SVs
There's nothing you can /need to do about it, aside from not worry.
// Jonathan Vanasco
| - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - - - - - - - - - - - - - -
| FindMeOn.com - The cure for Multiple Web Personality Disorder
| Web Identity Management and 3D Social Networking
| - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - - - - - - - - - - - - - -
| RoadSound.com - Tools For Bands, Stuff For Fans
| Collaborative Online Management And Syndication Tools
| - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - - - - - - - - - - - - - -
Re: Loading at startup to use shared memory
Posted by Kjetil Kjernsmo <kj...@opera.com>.
On Monday 06 November 2006 19:03, Jonathan Vanasco wrote:
> i've found the reports of shared/unshared memory to be completely
> useless
mmm, OK.
> the only way i've been able to bench memory usage is to run TOP in a
> terminal, and then kill/start apaches with new min_children
> settings, to find out the cost of the base memory , + each child.
Exactly what numbers are you then reading out of top then?
> then make 100-500 requests on each child to test the process growth.
OK.
> which version of apache are you using?
My test system is just upgraded to Ubuntu edgy, while our production
servers are Debian Sarge. This translates to 2.0.55 and 2.0.54,
respectively. Both systems run mp 2.0.2.
> are you using threads?
My test system uses mainly defaults, the production servers use preform
MPM, which I guess means no.
> on my app, which i've extensively profiles, i've got about 110MB of
> shared memory, and each child consumes 10-20 mb of unshared memory
Wow, that's very nice.
--
Kjetil Kjernsmo
Information Systems Developer
Opera Software ASA
Re: Loading at startup to use shared memory
Posted by Jonathan Vanasco <mo...@2xlp.com>.
i've found the reports of shared/unshared memory to be completely
useless
the only way i've been able to bench memory usage is to run TOP in a
terminal, and then kill/start apaches with new min_children settings,
to find out the cost of the base memory , + each child. then make
100-500 requests on each child to test the process growth.
which version of apache are you using? are you using threads?
on my app, which i've extensively profiles, i've got about 110MB of
shared memory, and each child consumes 10-20 mb of unshared memory
// Jonathan Vanasco
| - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - - - - - - - - - - - - - -
| FindMeOn.com - The cure for Multiple Web Personality Disorder
| Web Identity Management and 3D Social Networking
| - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - - - - - - - - - - - - - -
| RoadSound.com - Tools For Bands, Stuff For Fans
| Collaborative Online Management And Syndication Tools
| - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - - - - - - - - - - - - - -
Re: Loading at startup to use shared memory
Posted by Kjetil Kjernsmo <kj...@opera.com>.
On Tuesday 07 November 2006 17:17, Torsten Foertsch wrote:
> You can directly look at /proc/<PID>/smaps or use Linux::Smaps. I
> don't know what GTop does but I know how /proc/<PID>/smaps work.
> Also, don't do it too often. Reading from /proc/<PID>/smaps is really
> expensive particularly for large process sizes.
Wow. Yeah, that's really interesting! So, I started to dive into what
this means... I have a lot to learn here... Thanks, I'll be
investigating.
> Maybe I forgot to say in the docs, /proc/<PID>/smaps cannot determine
> if a page is shared or private if it is out of core (paged out).
> Hence to make your measurement accurate you must lock the parent
> apache in core or turn off swapping completely.
Heh, that's no problem, because our sysadmin insists that swapping
sucks, so there is no swap on those boxes...
Cheers,
Kjetil
--
Kjetil Kjernsmo
Information Systems Developer
Opera Software ASA
Re: Loading at startup to use shared memory
Posted by Torsten Foertsch <to...@gmx.net>.
On Tuesday 07 November 2006 13:28, Kjetil Kjernsmo wrote:
> > What operating system is it? If linux see
> > http://search.cpan.org/~drolsky/Apache-SizeLimit-0.9/lib/Apache/SizeL
> >imit.pm#linux
>
> Yep, it is GNU/Linux, kernel 2.6.17.
>
> That's very interesting. I hadn't read up on that, because I thought
> Apache2::Resource did the same thing, but yeah, I think that is
> something we want to look into. Also, you're saying that the
> $Apache2::SizeLimit::HOW_BIG_IS_IT is a better way to assess the shared
> memory usage then GTop?
You can directly look at /proc/<PID>/smaps or use Linux::Smaps. I don't know
what GTop does but I know how /proc/<PID>/smaps work. Also, don't do it too
often. Reading from /proc/<PID>/smaps is really expensive particularly for
large process sizes.
Maybe I forgot to say in the docs, /proc/<PID>/smaps cannot determine if a
page is shared or private if it is out of core (paged out). Hence to make
your measurement accurate you must lock the parent apache in core or turn off
swapping completely.
Locking the parent apache in memory can be done this way:
<Perl>
require 'syscall.ph';
require 'sys/mmap.ph';
0==syscall &SYS_mlockall, &MCL_CURRENT | &MCL_FUTURE or
die "ERROR: mlockall failed: $!\n";
</Perl>
or a bit simpler
<Perl>
0==syscall 152, 3 or
die "ERROR: mlockall failed: $!\n";
</Perl>
This prevents only the parent apache from being paged. The child processes
wont. But since the parent is always in core all shared pages are so.
@Dave, maybe you could add a word about the mlockall to the linux section of
Apache::SizeLimit?
Torsten
Re: Loading at startup to use shared memory
Posted by Kjetil Kjernsmo <kj...@opera.com>.
On Monday 06 November 2006 18:08, Torsten Foertsch wrote:
>
> What operating system is it? If linux see
> http://search.cpan.org/~drolsky/Apache-SizeLimit-0.9/lib/Apache/SizeL
>imit.pm#linux
Yep, it is GNU/Linux, kernel 2.6.17.
That's very interesting. I hadn't read up on that, because I thought
Apache2::Resource did the same thing, but yeah, I think that is
something we want to look into. Also, you're saying that the
$Apache2::SizeLimit::HOW_BIG_IS_IT is a better way to assess the shared
memory usage then GTop?
--
Kjetil Kjernsmo
Information Systems Developer
Opera Software ASA
Re: Loading at startup to use shared memory
Posted by Torsten Foertsch <to...@gmx.net>.
On Monday 06 November 2006 17:48, Kjetil Kjernsmo wrote:
> I'm working on improving the scalability and performance of our app. It
> is rather bad, and I noticed that not many minutes after restart, the
> amount of shared memory is down to 10% of the total memory (and I'm at
> least at 1/3 on my home Axkit::App::TABOO), and besides, it is bad from
> the start too.
>
> Obviously, there is a lot to gain here, since we run a lot of children
> across many backends.
>
> So, I pulled out Stas and Eric's book, which has a chapter about it.
> That's about mod_perl 1.0, but much of it I presume still is valid.
> Also, I found an example in
> http://perl.apache.org/docs/2.0/user/handlers/server.html#Startup_File
> which I tried.
>
> I have not used the example script of Stas and Eric, but instead insert
> some GTop stuff in a registry script that loads a small picture from
> the database. It doesn't really use a lot of the stuff I load at
> startup. Then I warn the numbers, to find them in the Apache log.
>
> Then, I have stared at the numbers with bewilderment.
>
> Without any startup.pl file:
> Size: 59346944 Shared: 6172672 Unshared: 53174272
>
> Then I set the startup.pl, but removed and included different modules,
> just to see what happens. The results are consistent, though:
> Size: 76079104 Shared: 5763072 Unshared: 70316032
> Size: 78835712 Shared: 5947392 Unshared: 72888320
> Size: 73920512 Shared: 4878336 Unshared: 69042176
>
> So, what I see is that consistently, the amount of shared memory goes
> down when I preload modules at server startup... Weird, huh?
> It is pretty clear that total memory per child must go up by preloading
> modules, but the idea was that the shared portion of it should be much
> greater... I mean, just the code, which is identical per child, should
> contribute to this...
>
> I've been monitoring top as the server runs too, and it seems consistent
> with these findings, also on other parts of the server. I guess we
> should rewrite a lot of stuff here, but I was hoping this would be a
> quick fix for some of our problems, as I think there is a lot in here
> that would reasonably be in shared memory pages. So, any ideas where I
> can start to understand why this isn't shared?
What operating system is it? If linux see
http://search.cpan.org/~drolsky/Apache-SizeLimit-0.9/lib/Apache/SizeLimit.pm#linux
Torsten