You are viewing a plain text version of this content. The canonical link for it is here.
Posted to modperl@perl.apache.org by "Richard F. Rebel" <rr...@whenu.com> on 2005/02/12 15:13:03 UTC
[Fwd: Re: /proc/*/statm, exactly what does "shared" mean?]
Hello Perrin and other modperl users,
I have forwarded an email from Hugh Dickens as a follow up on recent
discussion of mod_perl+apache forked children and copy-on-write pages.
As I had suspected, for our interest, the 'shared' figure represented by
TOP, ps, and other tools (based on /proc/<pid>/statm) does not and never
really has been a useful way to count how 'shared' our apache+mod_perl
processes are. It seems that in 2.2 and 2.4 that figure at least
included COW pages, but it was polluted with other types such as shared
libraries and mmaps etc. In recent 2.6 kernels it is more refined, but
does NOT include nor report COW pages.
Whoops, I keep on falling into a Linux only mindset. Let me the clear,
this is NOT necessarily the case with other un*x-es.
If I read between the lines, Hugh's message implies that counting COW's
may be reasonably significant performance hit, thus is why it's not
done.
Perrin, when I get a chance this week, I'll look into the GTop method I
have been using and see if it's based on /proc/<pid>/statm as well and
thus useless for this purpose.
I'll also continue my thread on the Linux kernel list to see if there is
a way to report on COW pages from the command line.
Best,
Richard F. Rebel
-------- Forwarded Message --------
> From: Hugh Dickins <hu...@veritas.com>
> To: Richard F. Rebel <rr...@whenu.com>
> Cc: linux-kernel@vger.kernel.org
> Subject: Re: /proc/*/statm, exactly what does "shared" mean?
> Date: Sat, 12 Feb 2005 13:06:39 +0000 (GMT)
> On Fri, 11 Feb 2005, Richard F. Rebel wrote:
> >
> > I can't seem to find clear documentation about the 'share' column
> > from /proc/<pid>/statm.
> >
> > Does this include pages that are shared with forked children marked as
> > copy-on-write?
> >
> > Does this only reflect libraries that are dynamically loaded? What
> > about shared memory segments/mmaps (ala shmat or mmmap)?
> >
> > If there is a place where I might find documentation that is more clear
> > beyond the proc.txt in the kernel docs and then man pages for procfs,
> > I'd welcome a pointer.
>
> You may not be entirely happy with this answer.
> It is a count of "pages of the process" which are "shared" in some sense.
> But precisely what that means has changed from time to time: depending on
> our perception of what we can safely afford the overhead of counting.
>
> You may want to look at fs/proc proc_pid_statm() source for the release
> of interest, and follow that back to see exactly what is being counted.
>
> Throughout 2.4 (and 2.2 too I think) it was the count of those pages
> instantiated in the process address space which currently have a page
> count greater than 1. That would include private pages shared with
> forked children, pages from the pagecache (including pages mapped
> from executable or library or shared memory or file mmap), those
> private pages which currently have swap allocated (so they're also
> in the swapcache), and any pages which transitorily have page count
> raised for whatever reason (they'd likely already be in one of the
> above categories). A ragbag of meanings, but that's all you can
> get from looking at page count.
>
> Counting up that not very meaningful number, at frequent intervals
> on large process address spaces, is a waste of valuable time.
>
> From 2.5.37 to 2.6.9, it's the total extent of file-backed areas
> (file including executable or library or shared memory) in the
> process address space: a total extent (in pagesize units),
> not a count of instantiated pages. Much quicker to calculate.
>
> But there were complaints about that, and a need to revert from
> total extent to count of instantiated pages.
>
> From 2.6.10 onwards, for the foreseeable future, it is the count
> of those pages instantiated in the process address space which are
> shared with a file (including executable or library or shared memory)
> i.e. those pages which are not anonymous, not private. That count
> does not include private pages shared with forked children, nor
> does it include private pages which happen to have swap allocated.
>
> Hugh
--
Richard F. Rebel <rr...@whenu.com>
WhenU.com
Re: measuring shared memory on linux (was [Fwd: Re: /proc/*/statm,
exactly what does "shared" mean?)]
Posted by "Richard F. Rebel" <rr...@whenu.com>.
On Mon, 2005-02-14 at 17:39 -0500, Perrin Harkins wrote:
> > Thanks for pursuing that issue at the linux kernel list, Richard. As you
> > have suggested it doesn't look very good.
>
> I want to add my thanks too.
You are welcome. I wish I wasn't the bearer of such bad news tho.
> Or on anything at all, since the size measurements they use include
> things that really shouldn't be counted as part of the process size.
>
> It seems that to tune properly you would need to send in requests with a
> benchmarking tool and keep adjusting MaxClients upward while watching
> the free memory on the box. That's really ugly. Maybe there's a way to
> count memory used by each process slowly but accurately which we could
> use during tuning at least.
>
> - Perrin
That is an ugly way, but probably the only way we have at this time.
I received further information on the linux kernel mailing list.
Basically we can apply a 'patch', okay, a *hack* that is very unlikely
to ever end up in the vanilla kernel and start maintaining it.
The way I see it, unless we patch/hack the kernel to count the
information we are interested in, we are SOL. If we do add it, expect a
significant performance hit and don't expect it to be in the vanilla
kernel.
I am going to *try* and take the code sample I was given and wedge it
into the right place on a machine sometime this week. If it works, I'll
let you know. Maybe we can convince someone on lkml that if we make a
way to turn this *feature* on or off with a sysctl and put it somewhere
else in /proc/<pid/* that they'll put it in the vanilla kernel. Then
someone with more of a clue than I have can maintain it :)
Best,
Richard
--
Richard F. Rebel
cat /dev/null > `tty`
Re: measuring shared memory on linux (was [Fwd: Re: /proc/*/statm,
exactly what does "shared" mean?)]
Posted by Stas Bekman <st...@stason.org>.
Perrin Harkins wrote:
> On Mon, 2005-02-14 at 18:45 -0500, Stas Bekman wrote:
>
>>that approach is not very practical if change your code base constantly.
>>Since you will have to retune things every time you change your code.
>
>
> I know, it's terrible, but it's all I've come up with so far. Maybe we
> need to rethink how the size-limiting modules work so that they can use
> something like the total amount of free physical RAM instead. I think
> that's sort of what you were thinking of too.
>
>
>>I'd rather see the kernel providing a new feature which tells us the exact
>>amount of memory used by a group of forked processes (swapped in and out).
>
>
> Me too, but I'm not holding my breath. We can find out how much total
> memory is free though, at least on Linux.
That particular information is far from being useful, since there is the
cache. On my (linux) machine I have 0MB of free memory and 400-500MB of
cached buffers. So on linux one needs to combine the two to get the
answer. (see the output of free(1) on linux to see what I'm talking about)
So what you are saying is that whatever the technique is to get that
remaining free memory, if we have it we could just set a limit on how much
"free" memory is available and start killing Apache procs, when that limit
is passed. I think that might just work. The roughest approach will be to
just kill the current process. The fine tuned one will be to maintain a
table of proc sizes as I've suggested before.
--
__________________________________________________________________
Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide ---> http://perl.apache.org
mailto:stas@stason.org http://use.perl.org http://apacheweek.com
http://modperlbook.org http://apache.org http://ticketmaster.com
Re: measuring shared memory on linux (was [Fwd: Re: /proc/*/statm,
exactly what does "shared" mean?)]
Posted by Salve J Nilsen <sj...@pvv.org>.
Suddenly, Perrin Harkins uttered:
> On Mon, 2005-02-14 at 18:45 -0500, Stas Bekman wrote:
>> that approach is not very practical if change your code base constantly.
>> Since you will have to retune things every time you change your code.
>
> I know, it's terrible, but it's all I've come up with so far.
Perhaps the CSA (Comprehensive System Accounting) and ELSA (Enhanced Linux
System Accounting) projects can do something for us? These provide patches
to the Linux kernel in order to improve process accounting.
CSA: http://oss.sgi.com/projects/csa/
ELSA: http://elsa.sourceforge.net/
Haven't looked at them closely, so this may be a shot in the dark... :-\
- Salve
--
#!/usr/bin/perl
sub AUTOLOAD{$AUTOLOAD=~/.*::(\d+)/;seek(DATA,$1,0);print# Salve Joshua Nilsen
getc DATA}$"="'};&{'";@_=unpack("C*",unpack("u*",':4@,$'.# <sj...@foo.no>
'2!--"5-(50P%$PL,!0X354UC-PP%/0\`'."\n"));eval "&{'@_'}"; __END__ is near! :)
Re: measuring shared memory on linux (was [Fwd: Re: /proc/*/statm,
exactly what does "shared" mean?)]
Posted by Perrin Harkins <pe...@elem.com>.
On Mon, 2005-02-14 at 18:45 -0500, Stas Bekman wrote:
> that approach is not very practical if change your code base constantly.
> Since you will have to retune things every time you change your code.
I know, it's terrible, but it's all I've come up with so far. Maybe we
need to rethink how the size-limiting modules work so that they can use
something like the total amount of free physical RAM instead. I think
that's sort of what you were thinking of too.
> I'd rather see the kernel providing a new feature which tells us the exact
> amount of memory used by a group of forked processes (swapped in and out).
Me too, but I'm not holding my breath. We can find out how much total
memory is free though, at least on Linux.
- Perrin
Re: measuring shared memory on linux (was [Fwd: Re: /proc/*/statm,
exactly what does "shared" mean?)]
Posted by Stas Bekman <st...@stason.org>.
Perrin Harkins wrote:
[...]
> It seems that to tune properly you would need to send in requests with a
> benchmarking tool and keep adjusting MaxClients upward while watching
> the free memory on the box. That's really ugly. Maybe there's a way to
> count memory used by each process slowly but accurately which we could
> use during tuning at least.
that approach is not very practical if change your code base constantly.
Since you will have to retune things every time you change your code.
I'd rather see the kernel providing a new feature which tells us the exact
amount of memory used by a group of forked processes (swapped in and out).
If we have this number we are gold. just maintain a table of absolute
process sizes and kill the biggest processes, when that total memory usage
number is bigger than the limit. i.e. limiting all procs at once, rather
on the process by process basis. So this will still provide a solid
protection from swapping.
--
__________________________________________________________________
Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide ---> http://perl.apache.org
mailto:stas@stason.org http://use.perl.org http://apacheweek.com
http://modperlbook.org http://apache.org http://ticketmaster.com
Re: measuring shared memory on linux (was [Fwd: Re: /proc/*/statm,
exactly what does "shared" mean?)]
Posted by Perrin Harkins <pe...@elem.com>.
On Sat, 2005-02-12 at 11:43 -0500, Stas Bekman wrote:
> Richard F. Rebel wrote:
> > Hello Perrin and other modperl users,
> >
> > I have forwarded an email from Hugh Dickens as a follow up on recent
> > discussion of mod_perl+apache forked children and copy-on-write pages.
>
> Thanks for pursuing that issue at the linux kernel list, Richard. As you
> have suggested it doesn't look very good.
I want to add my thanks too.
> At the same time the actual sharing is the same as before, so you still
> win from preloading things. It's just that you can't quantify it anymore.
This is especially unfortunate because preloading appears to make the
SIZE measurement increase, as does the use of mmap in things like
Cache::FastMmap. Before, we were assuming you could see how much memory
was truly being used by subtracting SHARE from SIZE, but it seems that
won't work.
> Once we get some "official" resolution we should update the tools like
> Apache-VMonitor to reflect that discovery, in order to not mislead users.
> More importantly it affects Apache::SizeLimit and Apache::GTopSizeLimit,
> which as the things are now, aren't working properly when someone limits
> on shared memory.
Or on anything at all, since the size measurements they use include
things that really shouldn't be counted as part of the process size.
It seems that to tune properly you would need to send in requests with a
benchmarking tool and keep adjusting MaxClients upward while watching
the free memory on the box. That's really ugly. Maybe there's a way to
count memory used by each process slowly but accurately which we could
use during tuning at least.
- Perrin
measuring shared memory on linux (was [Fwd: Re: /proc/*/statm, exactly
what does "shared" mean?)]
Posted by Stas Bekman <st...@stason.org>.
Richard F. Rebel wrote:
> Hello Perrin and other modperl users,
>
> I have forwarded an email from Hugh Dickens as a follow up on recent
> discussion of mod_perl+apache forked children and copy-on-write pages.
Thanks for pursuing that issue at the linux kernel list, Richard. As you
have suggested it doesn't look very good.
At the same time the actual sharing is the same as before, so you still
win from preloading things. It's just that you can't quantify it anymore.
Once we get some "official" resolution we should update the tools like
Apache-VMonitor to reflect that discovery, in order to not mislead users.
More importantly it affects Apache::SizeLimit and Apache::GTopSizeLimit,
which as the things are now, aren't working properly when someone limits
on shared memory.
Can others give us an update on other OSes? I know Gtop works on BSDs and
Solaris. Does it give the proper shared data over there? And the "manual"
methods used by Apache::SizeLimit?
--
__________________________________________________________________
Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide ---> http://perl.apache.org
mailto:stas@stason.org http://use.perl.org http://apacheweek.com
http://modperlbook.org http://apache.org http://ticketmaster.com