Re: [mp2] Perl Sections in .htaccess files leak memory

["Philippe M. Chiasson" <go...@ectoplasm.org>]

Rici Lake wrote:
> In the course of trying to figure out why mod_info does not reveal 
> directives added in PerlSections (see other email), I stumbled across 
> the following problem:
> 
> I created a simple seven-line .htaccess file, and did 100,000 requests 
> to a file in the corresponding directory with ab. All processes were 
> perfectly stable in memory consumption. Then I do the same thing with a 
> <Perl> section in the .htaccess file. 100,000 requests later my 12 
> processes had grown from 6.5M to 25.5M

Could you please post the 2 .htaccess files you've been using ?

> My analysis of this is that mod_perl's perl sections are *always* run 
> in the pconf pool, rather than running in the request pool as would be 
> normal for .htaccess files. Consequently, all directive parsing and 
> config merging will use the pconf pool, effectively resulting in a 
> permanent expansion of that pool.

Hrm, that's quite odd, as I looked over the code and AFAIK, <Perl> sections
in .htaccess will be running off parms->pool, and in the case of .htaccess
files, that's r->pool.  Can you tell me more about how you made the determination
that it was using the pconf ?

-- 
--------------------------------------------------------------------------------
Philippe M. Chiasson m/gozer\@(apache|cpan|ectoplasm)\.org/ GPG KeyID : 88C3A5A5
http://gozer.ectoplasm.org/     F9BF E0C2 480E 7680 1AE5 3631 CB32 A107 88C3A5A5

Re: [mp2] Perl Sections in .htaccess files leak memory

["Philippe M. Chiasson" <go...@ectoplasm.org>]

Rici Lake wrote:
> On 9-Sep-04, at 12:50 PM, Philippe M. Chiasson wrote:
> 
> 
>>Incredibly thorough bug reporting, many thanks.
> 
> 
> We aim to please ;)
> 
> 
>>You correctly identified a leak and can you give the following
>>patch a spin and let me know if it plugs it?
> 
> 
> The patch fixes the override problem but it seems to still be leaking 
> memory.

Fixing the override problem was a nice planned side-effect of this patch.

> I had to download the current CVS snapshot to apply it; I see that the 
> problems with building on FreeBSD have now been resolved, which is 
> great
> 
> Anyway, I patched the CVS snapshot to set override to OR_ALL rather 
> than the initial settings, so that my little .htaccess file could run 
> without generating errors. Then I ran roughly the same ab test as I did 
> before, on both the CVS snapshot and the CVS snapshot with your patch.
> 
> (ab -c 4 -n 100000)
> 
> This pretty well maxes out a 100Mb ethernet (even though the file 
> itself is tiny) so ab is not very helpful in terms of speed measures, 
> but running top at the same time lets me check out memory usage. The 
> children start out at 9.2 MB (size, not rss -- in all cases rss stays 
> pretty constantly at size - 3 MB). With the current snapshot, the 
> children end up being between 25 MB and 29.5 MB (and the server starts 
> to swap by the end of the test, my test machine only has 512 MB). With 
> your patch, this comes down to 18-23 MB. So the leakage was about 18 
> MB/child before your patch, and 11.3 MB/child with your patch.

Hrm, that's unfortunate it doesn't fully plug the leak.

> I'd say that's a big step forward: Override settings are obviously 
> important and the memory leakage is significantly reduced. (The slight 
> increase in response time was probably the result of not driving the 
> machine into swap.)
> 
> Where the rest of the memory is going, I don't know. Maybe the perl 
> environments are expanding?

That's a possibility I'll look into. I will be checking this leak fix
shortly along with another fix to the way package are unloaded and it
might help reduce this leaking as well.

Once I checked both fixes in, you could try your test once more and report
on the leaks it generates.

Thanks!

> R.
> 
> 

-- 
--------------------------------------------------------------------------------
Philippe M. Chiasson m/gozer\@(apache|cpan|ectoplasm)\.org/ GPG KeyID : 88C3A5A5
http://gozer.ectoplasm.org/     F9BF E0C2 480E 7680 1AE5 3631 CB32 A107 88C3A5A5

Re: [mp2] Perl Sections in .htaccess files leak memory

["Philippe M. Chiasson" <go...@ectoplasm.org>]

Incredibly thorough bug reporting, many thanks.

You correctly identified a leak and can you give the following
patch a spin and let me know if it plugs it?

Rici Lake wrote:
> On 8-Sep-04, at 5:24 PM, Philippe M. Chiasson wrote:
> 
>>Could you please post the 2 .htaccess files you've been using ?
>>
> 
> 
> rlake@freeb:/usr/local/www/data/foo$ cat .htaccess
> <perl>
> $AuthType = "basic";
> $AuthName = "foo";
> $AuthUserFile = "/usr/local/www/.htpasswd";
> $require = "valid-user";
> $order = "deny,allow";
> $allow = "from all";
> $satisfy = "any";
> </perl>
> 
> rlake@freeb:/usr/local/www/data/foo$ cat .htaccess.plain
> #<perl>
> AuthType basic
> AuthName foo
> AuthUserFile /usr/local/www/.htpasswd
> require valid-user
> order deny,allow
> allow from all
> satisfy any
> #</perl>
> 
>>Hrm, that's quite odd, as I looked over the code and AFAIK, <Perl> 
>>sections
>>in .htaccess will be running off parms->pool, and in the case of 
>>.htaccess
>>files, that's r->pool.  Can you tell me more about how you made the 
>>determination
>>that it was using the pconf ?
> 
> 
> Ok.
> 
> The handler for <Perl> (in modperl_cmd.c at line 418, 
> MP_CMD_SRV_DECLARE(perl)) appears to use the pool passed to it by the 
> config reader, which should be r->pool in an .htaccess file. But all it 
> does with it is concatenate the lines up to the closing </Perl> 
> directive into a single string, and use that to create a Perl directive 
> which it inserts into the command stream.
> 
> So that drops us into MP_CMD_SRV_DECLARE(perldo), immediately following 
> in the same file.
> 
> That function, afaics, starts by checking for options that might have 
> been set in the <Perl> directive itself (which it finds in 
> directive->data thanks to the <Perl> directive handler) and if there 
> are none, as in my simple test, uses Apache::PerlSections as the 
> handler and Apache::ReadConfig as the package. It then prepends the 
> original <Perl> section, now in directive->arg, with the package name 
> and executes it, at around line 538. If that all goes well, it does a 
> callback into the handler with modperl_callback.
> 
> So, taking a wild guess, I looked in Apache/PerlSections.pm, and indeed 
> that seems to be where the directives are created, by being pushed one 
> at a time into the array $self->directives. Finally self->post_config 
> gets called, and it in turn calls $self->server->add_config with the 
> directives.
> 
> According to the mapping in xs/Apache/ServerUtil/Apache__ServerUtil.h, 
> that ought to end up calling modperl_config_insert_server, which is 
> found in modperl_config.c. That function extracts pconf from the server 
> record, and calls modperl_config_insert with it, at which point the 
> configuration directives are fed into the Apache machinery.
> 
> Which is why I figured that it was using pconf.
> 
> I was actually just trying to figure out why directives were not being 
> placed into the configtree -- the reason is that modperl_config_insert 
> doesn't hook them into the tree. This doesn't matter for .htaccess 
> files; I just happened to notice that the .htaccess code path appeared 
> to be the same as the main config file code path.
> 
> So I tested it, and the test seemed to confirm my suspicions.
> 
> 

-- 
--------------------------------------------------------------------------------
Philippe M. Chiasson m/gozer\@(apache|cpan|ectoplasm)\.org/ GPG KeyID : 88C3A5A5
http://gozer.ectoplasm.org/     F9BF E0C2 480E 7680 1AE5 3631 CB32 A107 88C3A5A5