[RFC] Apache::SizeLimit on Linux

[Torsten Foertsch <to...@gmx.net>]

Hi,

now that we have Apache2::SizeLimit using Linux::Smaps the old functionality 
of accounting COW pages as shared is restored on linux 2.6. It should show 
for linux 2.4 and 2.6 the same behavior (can someone using 2.4 and 2.6 check 
this please? I have no 2.4 box available) since the algorithm computing the 
figures is the same.

Hence, the problems are the same when paging occurs. And these problems make 
it almost useless by default. The algorithm counting shared memory simply 
checks the reference count of each page table entry. If it is at least 2 the 
page is shared. If a page is not present (paged out) it is not counted at 
all.

Now what happens if the master apache is mostly paged out? Then there are only 
a few shared pages left. That makes SizeLimit terminate the process 
practically after each check.

I have found 3 solutions:

1. obviously: not to use swap space at all

2. mlockall(2) while the master is initializing. This means the master apache 
is always completely present in memory. The lock is not inherited by child 
processes, thus, the workers can be paged out.

3. patch linux to report shared swap space as well. I have not looked into 
this any further.

Would it be right to patch the mlockall(2) into SizeLimit.pm with an option to 
disable it?

This has the drawback that to mlockall a process you must be root. It could be 
done in a PostConfig handler that checks if MIN_SHARED or MAX_UNSHARED are 
set and executes the syscall only if yes. That would then also be the right 
place to check for the presence of /proc/PID/smaps. If only MAX_PROCESS_SIZE 
is set then checking /proc/PID/statm will also do and is a lot cheaper (on 
2.6).

I think a description of the behavior should be in the docs in any case.

Torsten

Re: [RFC] Apache::SizeLimit on Linux

[Stas Bekman <st...@stason.org>]

Torsten Foertsch wrote:
> Hi,
> 
> now that we have Apache2::SizeLimit using Linux::Smaps the old functionality 
> of accounting COW pages as shared is restored on linux 2.6. It should show 
> for linux 2.4 and 2.6 the same behavior (can someone using 2.4 and 2.6 check 
> this please? I have no 2.4 box available) since the algorithm computing the 
> figures is the same.
> 
> Hence, the problems are the same when paging occurs. And these problems make 
> it almost useless by default. The algorithm counting shared memory simply 
> checks the reference count of each page table entry. If it is at least 2 the 
> page is shared. If a page is not present (paged out) it is not counted at 
> all.
> 
> Now what happens if the master apache is mostly paged out? Then there are only 
> a few shared pages left. That makes SizeLimit terminate the process 
> practically after each check.
> 
> I have found 3 solutions:
> 
> 1. obviously: not to use swap space at all
> 
> 2. mlockall(2) while the master is initializing. This means the master apache 
> is always completely present in memory. The lock is not inherited by child 
> processes, thus, the workers can be paged out.
> 
> 3. patch linux to report shared swap space as well. I have not looked into 
> this any further.
> 
> Would it be right to patch the mlockall(2) into SizeLimit.pm with an option to 
> disable it?
> 
> This has the drawback that to mlockall a process you must be root. It could be 
> done in a PostConfig handler that checks if MIN_SHARED or MAX_UNSHARED are 
> set and executes the syscall only if yes. That would then also be the right 
> place to check for the presence of /proc/PID/smaps. If only MAX_PROCESS_SIZE 
> is set then checking /proc/PID/statm will also do and is a lot cheaper (on 
> 2.6).
> 
> I think a description of the behavior should be in the docs in any case.

I remember there was a long discussion on mlockall on the users list a few 
years ago. Please check the archives. I remember there were some issues 
with it, so we never adopted that approach.

Also shouldn't that functionality be abstracted into a module outside 
Apache2::? More people will be able to benefit from it and hopefully have 
a better testing/dev process.

I don't have 2.4 to test.

In any case, whatever the outcome of this thread is, I encourage you to 
document anything that you think needs to be documented. I haven't been 
using this tool for a while so I don't have much to say about the 
specifics of these changes.


-- 
__________________________________________________________________
Stas Bekman            JAm_pH ------> Just Another mod_perl Hacker
http://stason.org/     mod_perl Guide ---> http://perl.apache.org
mailto:stas@stason.org http://use.perl.org http://apacheweek.com
http://modperlbook.org http://apache.org   http://mailchannels.com

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@perl.apache.org
For additional commands, e-mail: dev-help@perl.apache.org