[mp2] Perl Sections in .htaccess files leak memory

[Rici Lake <ri...@ricilake.net>]

In the course of trying to figure out why mod_info does not reveal 
directives added in PerlSections (see other email), I stumbled across 
the following problem:

I created a simple seven-line .htaccess file, and did 100,000 requests 
to a file in the corresponding directory with ab. All processes were 
perfectly stable in memory consumption. Then I do the same thing with a 
<Perl> section in the .htaccess file. 100,000 requests later my 12 
processes had grown from 6.5M to 25.5M

My analysis of this is that mod_perl's perl sections are *always* run 
in the pconf pool, rather than running in the request pool as would be 
normal for .htaccess files. Consequently, all directive parsing and 
config merging will use the pconf pool, effectively resulting in a 
permanent expansion of that pool.


-- 
Report problems: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html

Re: [mp2] Perl Sections in .htaccess files leak memory

["Philippe M. Chiasson" <go...@ectoplasm.org>]

Rici Lake wrote:
> On 9-Sep-04, at 12:50 PM, Philippe M. Chiasson wrote:
> 
> 
>>Incredibly thorough bug reporting, many thanks.
> 
> 
> We aim to please ;)
> 
> 
>>You correctly identified a leak and can you give the following
>>patch a spin and let me know if it plugs it?
> 
> 
> The patch fixes the override problem but it seems to still be leaking 
> memory.

Fixing the override problem was a nice planned side-effect of this patch.

> I had to download the current CVS snapshot to apply it; I see that the 
> problems with building on FreeBSD have now been resolved, which is 
> great
> 
> Anyway, I patched the CVS snapshot to set override to OR_ALL rather 
> than the initial settings, so that my little .htaccess file could run 
> without generating errors. Then I ran roughly the same ab test as I did 
> before, on both the CVS snapshot and the CVS snapshot with your patch.
> 
> (ab -c 4 -n 100000)
> 
> This pretty well maxes out a 100Mb ethernet (even though the file 
> itself is tiny) so ab is not very helpful in terms of speed measures, 
> but running top at the same time lets me check out memory usage. The 
> children start out at 9.2 MB (size, not rss -- in all cases rss stays 
> pretty constantly at size - 3 MB). With the current snapshot, the 
> children end up being between 25 MB and 29.5 MB (and the server starts 
> to swap by the end of the test, my test machine only has 512 MB). With 
> your patch, this comes down to 18-23 MB. So the leakage was about 18 
> MB/child before your patch, and 11.3 MB/child with your patch.

Hrm, that's unfortunate it doesn't fully plug the leak.

> I'd say that's a big step forward: Override settings are obviously 
> important and the memory leakage is significantly reduced. (The slight 
> increase in response time was probably the result of not driving the 
> machine into swap.)
> 
> Where the rest of the memory is going, I don't know. Maybe the perl 
> environments are expanding?

That's a possibility I'll look into. I will be checking this leak fix
shortly along with another fix to the way package are unloaded and it
might help reduce this leaking as well.

Once I checked both fixes in, you could try your test once more and report
on the leaks it generates.

Thanks!

> R.
> 
> 

-- 
--------------------------------------------------------------------------------
Philippe M. Chiasson m/gozer\@(apache|cpan|ectoplasm)\.org/ GPG KeyID : 88C3A5A5
http://gozer.ectoplasm.org/     F9BF E0C2 480E 7680 1AE5 3631 CB32 A107 88C3A5A5

Re: [mp2] Perl Sections in .htaccess files leak memory

["Philippe M. Chiasson" <go...@ectoplasm.org>]

Incredibly thorough bug reporting, many thanks.

You correctly identified a leak and can you give the following
patch a spin and let me know if it plugs it?

Rici Lake wrote:
> On 8-Sep-04, at 5:24 PM, Philippe M. Chiasson wrote:
> 
>>Could you please post the 2 .htaccess files you've been using ?
>>
> 
> 
> rlake@freeb:/usr/local/www/data/foo$ cat .htaccess
> <perl>
> $AuthType = "basic";
> $AuthName = "foo";
> $AuthUserFile = "/usr/local/www/.htpasswd";
> $require = "valid-user";
> $order = "deny,allow";
> $allow = "from all";
> $satisfy = "any";
> </perl>
> 
> rlake@freeb:/usr/local/www/data/foo$ cat .htaccess.plain
> #<perl>
> AuthType basic
> AuthName foo
> AuthUserFile /usr/local/www/.htpasswd
> require valid-user
> order deny,allow
> allow from all
> satisfy any
> #</perl>
> 
>>Hrm, that's quite odd, as I looked over the code and AFAIK, <Perl> 
>>sections
>>in .htaccess will be running off parms->pool, and in the case of 
>>.htaccess
>>files, that's r->pool.  Can you tell me more about how you made the 
>>determination
>>that it was using the pconf ?
> 
> 
> Ok.
> 
> The handler for <Perl> (in modperl_cmd.c at line 418, 
> MP_CMD_SRV_DECLARE(perl)) appears to use the pool passed to it by the 
> config reader, which should be r->pool in an .htaccess file. But all it 
> does with it is concatenate the lines up to the closing </Perl> 
> directive into a single string, and use that to create a Perl directive 
> which it inserts into the command stream.
> 
> So that drops us into MP_CMD_SRV_DECLARE(perldo), immediately following 
> in the same file.
> 
> That function, afaics, starts by checking for options that might have 
> been set in the <Perl> directive itself (which it finds in 
> directive->data thanks to the <Perl> directive handler) and if there 
> are none, as in my simple test, uses Apache::PerlSections as the 
> handler and Apache::ReadConfig as the package. It then prepends the 
> original <Perl> section, now in directive->arg, with the package name 
> and executes it, at around line 538. If that all goes well, it does a 
> callback into the handler with modperl_callback.
> 
> So, taking a wild guess, I looked in Apache/PerlSections.pm, and indeed 
> that seems to be where the directives are created, by being pushed one 
> at a time into the array $self->directives. Finally self->post_config 
> gets called, and it in turn calls $self->server->add_config with the 
> directives.
> 
> According to the mapping in xs/Apache/ServerUtil/Apache__ServerUtil.h, 
> that ought to end up calling modperl_config_insert_server, which is 
> found in modperl_config.c. That function extracts pconf from the server 
> record, and calls modperl_config_insert with it, at which point the 
> configuration directives are fed into the Apache machinery.
> 
> Which is why I figured that it was using pconf.
> 
> I was actually just trying to figure out why directives were not being 
> placed into the configtree -- the reason is that modperl_config_insert 
> doesn't hook them into the tree. This doesn't matter for .htaccess 
> files; I just happened to notice that the .htaccess code path appeared 
> to be the same as the main config file code path.
> 
> So I tested it, and the test seemed to confirm my suspicions.
> 
> 

-- 
--------------------------------------------------------------------------------
Philippe M. Chiasson m/gozer\@(apache|cpan|ectoplasm)\.org/ GPG KeyID : 88C3A5A5
http://gozer.ectoplasm.org/     F9BF E0C2 480E 7680 1AE5 3631 CB32 A107 88C3A5A5

Re: [mp2] Perl Sections in .htaccess files leak memory

["Philippe M. Chiasson" <go...@ectoplasm.org>]

Rici Lake wrote:
> In the course of trying to figure out why mod_info does not reveal 
> directives added in PerlSections (see other email), I stumbled across 
> the following problem:
> 
> I created a simple seven-line .htaccess file, and did 100,000 requests 
> to a file in the corresponding directory with ab. All processes were 
> perfectly stable in memory consumption. Then I do the same thing with a 
> <Perl> section in the .htaccess file. 100,000 requests later my 12 
> processes had grown from 6.5M to 25.5M

Could you please post the 2 .htaccess files you've been using ?

> My analysis of this is that mod_perl's perl sections are *always* run 
> in the pconf pool, rather than running in the request pool as would be 
> normal for .htaccess files. Consequently, all directive parsing and 
> config merging will use the pconf pool, effectively resulting in a 
> permanent expansion of that pool.

Hrm, that's quite odd, as I looked over the code and AFAIK, <Perl> sections
in .htaccess will be running off parms->pool, and in the case of .htaccess
files, that's r->pool.  Can you tell me more about how you made the determination
that it was using the pconf ?

-- 
--------------------------------------------------------------------------------
Philippe M. Chiasson m/gozer\@(apache|cpan|ectoplasm)\.org/ GPG KeyID : 88C3A5A5
http://gozer.ectoplasm.org/     F9BF E0C2 480E 7680 1AE5 3631 CB32 A107 88C3A5A5