mod_perl lazy loading modules with eval in custom lib

[Christiaan Kras <c....@pcc-online.net>]

Hi,

My webhost allows me to use mod_perl (Apache 2 server by the way, Fedora 
distro). And I'm glad I can because it speeds up my perl scripts a lot.
   
I've created a website that's based on CGI::Application, 
Template-Toolkit and DBIx::Class. All works fine.
Those modules, including my own, are in a custom lib. Done so with use 
lib qw(etc etc etc) and just to be sure I've set PERL5LIB.
I've been running it under ModPerl::Registry and ModPerl::PerlRun. Both 
give me the same, infrequent errors.

Modules like Template::Plugins use lazy loading. I load 
Template::Plugin::Date by default (TT is in my custom lib).
The lazy loading happens with a require in an eval block. But somehow, 
my @INC has been reset when going through that eval() part.
Same thing happens with Excel::Template. This modules does the same, but 
in BEGIN blocks.

My scripts don't die. The eval() parts sometimes fail (because it can't 
find the modules). But the rest keeps on working.
I've tried lots of things and have searched all over internet. But I 
can't find a solution. I have no access to httpd.conf so I can't set my 
custom lib at server startup

The problem only happens when a module is loaded in an eval(). Somehow 
my @INC is getting reset. And I don't want that to happen :-)

I've searched perl.apache.org and searched all over internet to find a 
solution. But I couldn't find any.

I'm 100% sure it happens in the modules that use lazy loading 
(eval(require "module.pm")). Somehow @INC gets reset. My errorlog then 
states it can't find the required module. Weird thing is, it happens 
infrequently.

Anyone that had a similar problem and found a solution? Thanks in 
advance :-)

-- 

Christiaan Kras

Re: mod_perl lazy loading modules with eval in custom lib

[Christiaan Kras <c....@pcc-online.net>]

Perrin Harkins schreef:
> On 7/11/07, Christiaan Kras <c....@pcc-online.net> wrote:
>> By the way, I've used push() to add my path to @INC before. But that
>> didn't work out.
>
> It should work fine.  The reason people typically use unshift for this
> is that it lets modules in your local path override ones in the system
> path.
>
> - Perrin
>

I see. Well, it still seems to work but I'll get back to the list if it 
isn't. Although I'm sure now all is well.

-- 


Christiaan Kras

Re: mod_perl lazy loading modules with eval in custom lib

[Perrin Harkins <pe...@elem.com>]

On 7/11/07, Christiaan Kras <c....@pcc-online.net> wrote:
> By the way, I've used push() to add my path to @INC before. But that
> didn't work out.

It should work fine.  The reason people typically use unshift for this
is that it lets modules in your local path override ones in the system
path.

- Perrin

Re: mod_perl lazy loading modules with eval in custom lib

[Christiaan Kras <c....@pcc-online.net>]

Perrin Harkins schreef:
> On 7/11/07, Christiaan Kras <c....@pcc-online.net> wrote:
>> Thanks. I will try that. So basicly, the modules that are loaded at the
>> first request will stay in memory, but @INC will get reset?
>
> Yes.  It's a simulation of CGI behavior.  The only reason your CGI
> scripts don't have this problem is that they execute the "use lib" on
> every request, since they are compiling from scratch every time.
>
I've read about it. I was sure I checked it.
>> Wow. I never noticed it. In my app I have runmode that shows me %ENV and
>> @INC. At the first request my custom path was in there. And at the
>> second request it wasn't. But after a few more requests it's still
>> there. Weird behaviour?
>
> You're probably just seeing a new child process compile the script for
> the first time.
>
>> Just a quick note: unshift removes the first record from the array.
>
> No, it adds to the beginning of the array.
>
> - Perrin
>
Oops, my bad. *ashamed*


Anyway. I've tested it, and now in the runmode that show's me @INC it 
keeps the path I've set. I've tried a number of Excel exports as well 
and none of them were faulty. However, I will keep a look at my logfile. 
But I hope this solves the problem.

By the way, I've used push() to add my path to @INC before. But that 
didn't work out. Was that because push() puts the path at the end of 
@INC? Shouldn't really matter should it? Although it's possible I used 
it after loading my modules (not in the startup sub).

For now I seem to be helped. Thanks again :-).

-- 


Christiaan Kras

Re: mod_perl lazy loading modules with eval in custom lib

[Perrin Harkins <pe...@elem.com>]

On 7/11/07, Christiaan Kras <c....@pcc-online.net> wrote:
> Thanks. I will try that. So basicly, the modules that are loaded at the
> first request will stay in memory, but @INC will get reset?

Yes.  It's a simulation of CGI behavior.  The only reason your CGI
scripts don't have this problem is that they execute the "use lib" on
every request, since they are compiling from scratch every time.

> Wow. I never noticed it. In my app I have runmode that shows me %ENV and
> @INC. At the first request my custom path was in there. And at the
> second request it wasn't. But after a few more requests it's still
> there. Weird behaviour?

You're probably just seeing a new child process compile the script for
the first time.

> Just a quick note: unshift removes the first record from the array.

No, it adds to the beginning of the array.

- Perrin

Re: mod_perl lazy loading modules with eval in custom lib

[Clinton Gormley <cl...@traveljury.com>]

> Wow. I never noticed it. In my app I have runmode that shows me %ENV and 
> @INC. At the first request my custom path was in there. And at the 
> second request it wasn't. But after a few more requests it's still 
> there. Weird behaviour?

All that means is that you're hitting a new child for the first time.

> 
> Anyway, thanks. I'll try it later on. I'll post again if it doesn't help.
> 
> Just a quick note: unshift removes the first record from the array. But 
> what if my path isn't in @INC at a second request. That would remove a 
> normal path that's always around wouldn't it? Guess I'll just have to 
> try and see :-). Thanks.

You're thinking of shift :)

unshift is the opposite - it adds the new path onto the front of @INC.

Clint
> 

Re: mod_perl lazy loading modules with eval in custom lib

[Christiaan Kras <c....@pcc-online.net>]

Perrin Harkins schreef:
> On 7/11/07, Christiaan Kras <c....@pcc-online.net> wrote:
>> I add to @INC with " use lib qw(/my/path) ". All the other modules I
>> load continue to work fine. Just those that use lazy loading fail
>> infrequently.
>
> Okay, I think I see the problem.  When you change @INC during a
> request, the change only lasts for the length of that one request:
> http://perl.apache.org/docs/2.0/user/coding/coding.html#Request_localized_Globals 
>
>
> Because you do a "use lib", @INC is only set the first time you run
> this script.  Change it to this:
> unshift @INC, qw(/my/path/);
>
> - Perrin
>
Thanks. I will try that. So basicly, the modules that are loaded at the 
first request will stay in memory, but @INC will get reset?

...testing...

Wow. I never noticed it. In my app I have runmode that shows me %ENV and 
@INC. At the first request my custom path was in there. And at the 
second request it wasn't. But after a few more requests it's still 
there. Weird behaviour?

Anyway, thanks. I'll try it later on. I'll post again if it doesn't help.

Just a quick note: unshift removes the first record from the array. But 
what if my path isn't in @INC at a second request. That would remove a 
normal path that's always around wouldn't it? Guess I'll just have to 
try and see :-). Thanks.

-- 

Christiaan Kras

Re: mod_perl lazy loading modules with eval in custom lib

[Perrin Harkins <pe...@elem.com>]

On 7/11/07, Christiaan Kras <c....@pcc-online.net> wrote:
> I add to @INC with " use lib qw(/my/path) ". All the other modules I
> load continue to work fine. Just those that use lazy loading fail
> infrequently.

Okay, I think I see the problem.  When you change @INC during a
request, the change only lasts for the length of that one request:
http://perl.apache.org/docs/2.0/user/coding/coding.html#Request_localized_Globals

Because you do a "use lib", @INC is only set the first time you run
this script.  Change it to this:
unshift @INC, qw(/my/path/);

- Perrin

Re: mod_perl lazy loading modules with eval in custom lib

[Christiaan Kras <c....@pcc-online.net>]

Perrin Harkins schreef:
> On 7/7/07, Christiaan Kras <c....@pcc-online.net> wrote:
>> Because in my error messages it clearly says it can't find the modules
>> in @INC. Then it shows all the paths in @INC and my custom lib isn't set
>> there, instead, @INC is at it's original state.
>
> Okay, you said you don't add to @INC at startup.  How do you adjust
> @INC?  Is it possible you have a code path that doesn't set @INC?  Is
> it possible that some of your code is still running under CGI?
>
> - Perrin
>
I add to @INC with " use lib qw(/my/path) ". All the other modules I 
load continue to work fine. Just those that use lazy loading fail 
infrequently.
I've build it in a way that it will run under CGI as well as mod-perl 
(using perl-script handler). For a temporary solution I've added a CGI 
script that's exactly the same as the perl script, but run under CGI, to 
generate my Excel files without error. Not a very comforting solution, 
but for now it works. Although I would love to know the real solution.

Thanks.


-- 
Christiaan Kras

Re: mod_perl lazy loading modules with eval in custom lib

[Perrin Harkins <pe...@elem.com>]

On 7/7/07, Christiaan Kras <c....@pcc-online.net> wrote:
> Because in my error messages it clearly says it can't find the modules
> in @INC. Then it shows all the paths in @INC and my custom lib isn't set
> there, instead, @INC is at it's original state.

Okay, you said you don't add to @INC at startup.  How do you adjust
@INC?  Is it possible you have a code path that doesn't set @INC?  Is
it possible that some of your code is still running under CGI?

- Perrin

Re: mod_perl lazy loading modules with eval in custom lib

[Christiaan Kras <c....@pcc-online.net>]

Yes,

Because in my error messages it clearly says it can't find the modules 
in @INC. Then it shows all the paths in @INC and my custom lib isn't set 
there, instead, @INC is at it's original state.

Christiaan Kras

Perrin Harkins schreef:
> On 7/6/07, Christiaan Kras <c....@pcc-online.net> wrote:
>> I'm 100% sure it happens in the modules that use lazy loading
>> (eval(require "module.pm")). Somehow @INC gets reset.
>
> Are you sure @INC is getting reset?  Sometimes people have this
> problem because the user that apache runs as (usually "nobody")
> doesn't have permissions to access the directories in @INC.  At
> startup, they get loaded as root, so this isn't an issue at that time.
>
> - Perrin
>

Re: mod_perl lazy loading modules with eval in custom lib

[Perrin Harkins <pe...@elem.com>]

On 7/6/07, Christiaan Kras <c....@pcc-online.net> wrote:
> I'm 100% sure it happens in the modules that use lazy loading
> (eval(require "module.pm")). Somehow @INC gets reset.

Are you sure @INC is getting reset?  Sometimes people have this
problem because the user that apache runs as (usually "nobody")
doesn't have permissions to access the directories in @INC.  At
startup, they get loaded as root, so this isn't an issue at that time.

- Perrin