You are viewing a plain text version of this content. The canonical link for it is here.
Posted to modperl@perl.apache.org by Torsten Foertsch <to...@gmx.net> on 2005/08/07 20:37:34 UTC

[PATCH] Apache2::SizeLimit on Linux

Hi,

the attached patch allows Apache2::SizeLimit to use the new /proc/PID/smaps 
instead of /proc/PID/statm. This takes into account copy-on-write pages when 
counting shared memory.

The patch looks if /proc/PID/smaps exists and if Linux::Smaps is installed. If 
not it uses the old /proc/PID/statm. Hence, it is compatible with old 
linuxes.

/proc/PID/smaps exists by now in the mm-series of the linux kernel. I hope the 
patch will take its way into the vanilla kernel in the not so distant future.

The following handler shows the effect:

<Perl>
require Apache2::SizeLimit;
package X;
use strict;
use Apache2::RequestRec ();
use Apache2::RequestIO ();
use Apache2::Const -compile=>qw(OK);

my $x="a"x(1024*1024);

sub handler {
  my $r=shift;
  my $args=$r->args;
  my ($size, $shared)=$Apache2::SizeLimit::HOW_BIG_IS_IT->();
  $x=~tr/a/b/ if( $args );
  my ($size2, $shared2)=$Apache2::SizeLimit::HOW_BIG_IS_IT->();
  $r->content_type('text/plain');
  $r->print("1: size=$size shared=$shared\n");
  $r->print("2: size=$size2 shared=$shared2\n");
  return Apache2::Const::OK;
}
</Perl>

<Location /X>
SetHandler modperl
PerlResponseHandler X
</Location>

The parent apache allocates a megabyte for the string in $x. The tr-command 
then overwrites all "a" with "b" if the handler is called with an argument.
This write is done in place, thus, the process size doesn't change. Only $x is 
not shared anymore by means of COW between the parent and the child.

When Smaps are available curl shows:

  r2@s93:~/work/mp2> curl http://localhost:8181/X?1
  1: size=13452 shared=7456
  2: size=13452 shared=6432

The shared memory has lost 1024 kB.

Without Smaps it says:

  r2@s93:~/work/mp2> curl http://localhost:8181/X?1
  1: size=13052 shared=3628
  2: size=13052 shared=3636

One can see the kernel lies about the shared memory. It simply doesn't count 
COW pages as shared.

Both experiments are done with a Suse9.3, apache 2.0.54, mod_perl 2.0.1 and 
Linux::Smaps 0.01.

The first one uses linux 2.6.13-rc4-mm1 (with a little patch that will go into 
the next mm-version, see 
http://marc.theaimsgroup.com/?l=linux-kernel&m=112335328222533&w=2), the 
second with Suses default kernel (2.6.11.4-20a-default).

Torsten

Re: [PATCH] Apache2::SizeLimit on Linux

Posted by Torsten Foertsch <to...@gmx.net>.
On Friday 19 August 2005 18:59, Philip M. Gollucci wrote:
> Can you combine your 2 e-mail patches into one patch set in one  e-mail
> to make the life of who ever does easier?  At least I think this is the
> second one.

Here it is.

Torsten

Re: [PATCH] Apache2::SizeLimit on Linux

Posted by "Philip M. Gollucci" <pg...@p6m7g8.com>.
Torsten Foertsch wrote:
> On Monday 08 August 2005 01:30, Stas Bekman wrote:
> It corrects some podchecker errors and warnings as well.
Hi,

Great work.

I'm not going to commit it as I don't have a linux bed readily available 
to test on... Pretty sad for a guy that works with LAMP stacks :)

I'm +1 for it though.

Can you combine your 2 e-mail patches into one patch set in one  e-mail 
to make the life of who ever does easier?  At least I think this is the 
second one.

Thanks!

-- 
END
------------------------------------------------------------
     What doesn't kill us can only make us stronger.
                 Nothing is impossible.
				
Philip M. Gollucci (pgollucci@p6m7g8.com) 301.254.5198
Consultant / http://p6m7g8.net/Resume/
Senior Developer / Liquidity Services, Inc.
   http://www.liquidityservicesinc.com
        http://www.liquidation.com
        http://www.uksurplus.com
        http://www.govliquidation.com
        http://www.gowholesale.com


Re: [PATCH] Apache2::SizeLimit on Linux

Posted by Torsten Foertsch <to...@gmx.net>.
On Monday 08 August 2005 01:30, Stas Bekman wrote:
> Torsten Foertsch wrote:
> > Hi,
> >
> > the attached patch allows Apache2::SizeLimit to use the new
> > /proc/PID/smaps instead of /proc/PID/statm. This takes into account
> > copy-on-write pages when counting shared memory.
>
> Torsten++!
>
> Could you please submit a doc patch for Apache2/SizeLimit.pod
> http://svn.apache.org/viewcvs.cgi/perl/modperl/docs/trunk/src/docs/2.0/api/
>Apache2/SizeLimit.pod?rev=159831&view=log explaining the fact that shared
> memory calculation is normally wrong under linux kernel 2.6 and how to fix
> that situation? (below)

see attachment

It corrects some podchecker errors and warnings as well.

> > The patch looks if /proc/PID/smaps exists and if Linux::Smaps is
> > installed. If not it uses the old /proc/PID/statm. Hence, it is
> > compatible with old linuxes.
> >
> > /proc/PID/smaps exists by now in the mm-series of the linux kernel. I
> > hope the patch will take its way into the vanilla kernel in the not so
> > distant future.
>
> [...]
>
> > The first one uses linux 2.6.13-rc4-mm1 (with a little patch that will go
> > into the next mm-version, see
> > http://marc.theaimsgroup.com/?l=linux-kernel&m=112335328222533&w=2), the
> > second with Suses default kernel (2.6.11.4-20a-default).
>
> Do you know when the mm-series will become a part of the mainstream branch?

MM (Andrew Morton) says: "Yes, I expect we'll merge it up.". Since he is the 
maintainer of the 2.6 vanilla linux I think it can be expected rather sooner 
than later.

Torsten

Re: [PATCH] Apache2::SizeLimit on Linux

Posted by Stas Bekman <st...@stason.org>.
Torsten Foertsch wrote:
> Hi,
> 
> the attached patch allows Apache2::SizeLimit to use the new /proc/PID/smaps 
> instead of /proc/PID/statm. This takes into account copy-on-write pages when 
> counting shared memory.

Torsten++!

Could you please submit a doc patch for Apache2/SizeLimit.pod
http://svn.apache.org/viewcvs.cgi/perl/modperl/docs/trunk/src/docs/2.0/api/Apache2/SizeLimit.pod?rev=159831&view=log
explaining the fact that shared memory calculation is normally wrong under 
linux kernel 2.6 and how to fix that situation? (below)

> The patch looks if /proc/PID/smaps exists and if Linux::Smaps is installed. If 
> not it uses the old /proc/PID/statm. Hence, it is compatible with old 
> linuxes.
> 
> /proc/PID/smaps exists by now in the mm-series of the linux kernel. I hope the 
> patch will take its way into the vanilla kernel in the not so distant future.
[...]
> The first one uses linux 2.6.13-rc4-mm1 (with a little patch that will go into 
> the next mm-version, see 
> http://marc.theaimsgroup.com/?l=linux-kernel&m=112335328222533&w=2), the 
> second with Suses default kernel (2.6.11.4-20a-default).

Do you know when the mm-series will become a part of the mainstream branch?

-- 
__________________________________________________________________
Stas Bekman            JAm_pH ------> Just Another mod_perl Hacker
http://stason.org/     mod_perl Guide ---> http://perl.apache.org
mailto:stas@stason.org http://use.perl.org http://apacheweek.com
http://modperlbook.org http://apache.org   http://ticketmaster.com

Re: [PATCH] Apache2::SizeLimit on Linux

Posted by "Philip M. Gollucci" <pg...@p6m7g8.com>.
Torsten Foertsch wrote:
  > --- mod_perl-2.0.1/lib/Apache2/SizeLimit.pm~	2005-04-26 
20:58:44.000000000 +0200
> +++ mod_perl-2.0.1/lib/Apache2/SizeLimit.pm	2005-08-07 17:56:00.691361040 +0200
> @@ -56,7 +56,11 @@
>  
>      } elsif (LINUX) {
>  
> -        $HOW_BIG_IS_IT = \&linux_size_check;
> +        if ( eval { require Linux::Smaps } and Linux::Smaps->new($$) ) {
> +	    $HOW_BIG_IS_IT = \&linux_smaps_size_check;
> +	} else {
> +	    $HOW_BIG_IS_IT = \&linux_size_check;
> +	}
>  
>      } elsif (BSD_LIKE) {
>  
> @@ -85,6 +89,11 @@
>      }
>  }
>  
> +sub linux_smaps_size_check {
> +    my $s=Linux::Smaps->new($$)->all;
> +    return ($s->size, $s->shared_clean+$s->shared_dirty);
> +}
> +
>  # return process size (in KB)
>  sub linux_size_check {
>      my($size, $resident, $share) = (0, 0, 0);
It looks sane to me, but I don't have a Linux comp to test on yet.  I 
might in a week or so; but I wouldn't be able to commit it until then.


-- 
END
------------------------------------------------------------
     What doesn't kill us can only make us stronger.
                 Nothing is impossible.
				
Philip M. Gollucci (pgollucci@p6m7g8.com) 301.254.5198
Consultant / http://p6m7g8.net/Resume/
Senior Developer / Liquidity Services, Inc.
   http://www.liquidityservicesinc.com
        http://www.liquidation.com
        http://www.uksurplus.com
        http://www.govliquidation.com
        http://www.gowholesale.com


Re: [PATCH] Apache2::SizeLimit on Linux

Posted by Perrin Harkins <pe...@elem.com>.
On Sun, 2005-08-07 at 20:37 +0200, Torsten Foertsch wrote:
> the attached patch allows Apache2::SizeLimit to use the new /proc/PID/smaps 
> instead of /proc/PID/statm. This takes into account copy-on-write pages when 
> counting shared memory.

Thanks, Torsten.  I saw your module hit CPAN and was thinking we should
incorporate this into SizeLimit, and now here's the patch!

- Perrin