You are viewing a plain text version of this content. The canonical link for it is here.
Posted to modperl@perl.apache.org by Ben Wilder <be...@levexis.com> on 2006/08/31 10:32:01 UTC
Reloading .so files and .pm files
Hi all,
A Mod_perl question if I may!
Please shout if I haven't made my situation clear
Application behaviour:
Step 1.
The user requests a page and a Mod_perl script is called.
Step 2.
Based on user input, the script will dynamically 'require' a particular
module. This module calls a method from a .so file (The .so file has been
compiled from C and contains a number of statistical math calls best done in
C - this was then wrapped using SWIG for perl use)
Step 3.
The method call from the .so file returns a value to the module and then the
calling script. This drives logic in the originating Mod_perl script.
Step 4.
Content is returned to the user
Question:
What with the nature of the statistics I need to call from the compiled
shared objects, I am recompiling the .so files about twice a day (as
situations change). I need to update the .so files and their corresponding
.pm module files.
Does anyone have any suggestions as to how I could expire the old ones the
Mod_perl processes would be using get Mod_perl to pick up the newly updated
.so and .pm files?
I would have a directory of roughly 100 .so files and their corresponding
.pm files (generated by SWIG) that would be dynamically 'required' by
calling scripts, about half of these 100 shared objects would be recompiled
about twice a day!
Any thoughts much appreciated!
Regards
Ben
Re: Reloading .so files and .pm files
Posted by "Philip M. Gollucci" <pg...@p6m7g8.com>.
> ## Or this is already in PERL5INC some other way already
^^^ LIB
d'oh
Thats why you shouldn't code in thunderbird :)
--
------------------------------------------------------------------------
Philip M. Gollucci (pgollucci@p6m7g8.com) 323.219.4708
Consultant / http://p6m7g8.net/Resume/resume.shtml
Senior Software Engineer - TicketMaster - http://ticketmaster.com
1024D/A79997FA F357 0FDD 2301 6296 690F 6A47 D55A 7172 A799 97F
"In all that I've done wrong I know I must have done something right to
deserve a hug every morning and butterfly kisses at night."
__ ___ ___ ____ __
/ |/ /_ __/ __/ __ \/ /
/ /|_/ / // /\ \/ /_/ / /__
/_/ /_/\_, /___/\___\_\___/
<___/
RE: Reloading .so files and .pm files
Posted by Ben Wilder <be...@levexis.com>.
Hi Philip,
Just a note to thank you for taking the time to put together an example.
It's very much appreciated, and has taught me a fair bit. Your comments,
examples and suggestions have enabled me to create a working prototype.
I have a few long evenings with the Mod_perl docs I believe! :)
Regards,
Ben
-----Original Message-----
From: Philip M. Gollucci [mailto:pgollucci@p6m7g8.com]
Sent: 04 September 2006 10:01
To: ben.wilder@levexis.com
Cc: modperl@perl.apache.org
Subject: Re: Reloading .so files and .pm files
Ben Wilder wrote:
> 1) Your handler example is fantastic, however in my application I need to
> set the $pkg (the package to be dynamically included) from some logic in
> another script, so the process may go as follows:
> User request -> main script -> database call to get $pkg name - >
> dynamically call the package (as per your example).
See attached script.pl, Util.pm, and httpd.conf which will runs under the
ModPerl::Registry
handler (handlers are better though :)). The functionality is now in a
library and can be called
from anywhere.
Your variables need to be scoped to a FUNCTION via
'my' (or re-initialized properly on each request) This scoping is the
solution to your caching problem.
You need to read up on the "Core" modperl principles and what it actually
does with your code and how it caches it.
http://perl.apache.org/docs/1.0/guide/porting.html
While for 1.x the same principles apply to 2.x
> If my thinking is correct (and I'm a little new to Mod_perl!) the handler
> example you kindly wrote will be called when the /foo path is visited,
> however I will be needing to deliver this from a number of places (and
will
> not be able to rely on paths in the url) and would IDEALLY call the
dynamic
> functionality from within a script, rather than having it as a handler. Is
> there a way to do this and still not have the cache problem?
Thats correct, and see the attachments.
> 2) What is the best way to install the libapreq objects, can the same
> functionality be got through the Apache2 classes?
http://httpd.apache.org/apreq
download the lastest 2.08 un zip/tar and read the INSTALL file.
Apache2::* classes are wrapper around the APR::* classes (similar but not
always directly equivalent)
In general though both can get you what you want.
(attachments use Apache2::Request instead of APR::Request::Apache2)
http://httpd.apache.org/apreq/docs/libapreq2/modules.html
HTH
------------------------------------------------------------------------
Philip M. Gollucci (pgollucci@p6m7g8.com) 323.219.4708
Consultant / http://p6m7g8.net/Resume/resume.shtml
Senior Software Engineer - TicketMaster - http://ticketmaster.com
1024D/A79997FA F357 0FDD 2301 6296 690F 6A47 D55A 7172 A799 97F
"In all that I've done wrong I know I must have done something right to
deserve a hug every morning and butterfly kisses at night."
__ ___ ___ ____ __
/ |/ /_ __/ __/ __ \/ /
/ /|_/ / // /\ \/ /_/ / /__
/_/ /_/\_, /___/\___\_\___/
<___/
Re: Reloading .so files and .pm files
Posted by "Philip M. Gollucci" <pg...@p6m7g8.com>.
Ben Wilder wrote:
> 1) Your handler example is fantastic, however in my application I need to
> set the $pkg (the package to be dynamically included) from some logic in
> another script, so the process may go as follows:
> User request -> main script -> database call to get $pkg name - >
> dynamically call the package (as per your example).
See attached script.pl, Util.pm, and httpd.conf which will runs under the ModPerl::Registry
handler (handlers are better though :)). The functionality is now in a library and can be called
from anywhere.
Your variables need to be scoped to a FUNCTION via
'my' (or re-initialized properly on each request) This scoping is the solution to your caching problem.
You need to read up on the "Core" modperl principles and what it actually does with your code and how it caches it.
http://perl.apache.org/docs/1.0/guide/porting.html
While for 1.x the same principles apply to 2.x
> If my thinking is correct (and I'm a little new to Mod_perl!) the handler
> example you kindly wrote will be called when the /foo path is visited,
> however I will be needing to deliver this from a number of places (and will
> not be able to rely on paths in the url) and would IDEALLY call the dynamic
> functionality from within a script, rather than having it as a handler. Is
> there a way to do this and still not have the cache problem?
Thats correct, and see the attachments.
> 2) What is the best way to install the libapreq objects, can the same
> functionality be got through the Apache2 classes?
http://httpd.apache.org/apreq
download the lastest 2.08 un zip/tar and read the INSTALL file.
Apache2::* classes are wrapper around the APR::* classes (similar but not always directly equivalent)
In general though both can get you what you want.
(attachments use Apache2::Request instead of APR::Request::Apache2)
http://httpd.apache.org/apreq/docs/libapreq2/modules.html
HTH
------------------------------------------------------------------------
Philip M. Gollucci (pgollucci@p6m7g8.com) 323.219.4708
Consultant / http://p6m7g8.net/Resume/resume.shtml
Senior Software Engineer - TicketMaster - http://ticketmaster.com
1024D/A79997FA F357 0FDD 2301 6296 690F 6A47 D55A 7172 A799 97F
"In all that I've done wrong I know I must have done something right to
deserve a hug every morning and butterfly kisses at night."
__ ___ ___ ____ __
/ |/ /_ __/ __/ __ \/ /
/ /|_/ / // /\ \/ /_/ / /__
/_/ /_/\_, /___/\___\_\___/
<___/
RE: Reloading .so files and .pm files
Posted by Ben Wilder <be...@levexis.com>.
Hi Philip,
Thanks again, your reply has been very useful. Just a few more questions if
I may though!
1) Your handler example is fantastic, however in my application I need to
set the $pkg (the package to be dynamically included) from some logic in
another script, so the process may go as follows:
User request -> main script -> database call to get $pkg name - >
dynamically call the package (as per your example).
If my thinking is correct (and I'm a little new to Mod_perl!) the handler
example you kindly wrote will be called when the /foo path is visited,
however I will be needing to deliver this from a number of places (and will
not be able to rely on paths in the url) and would IDEALLY call the dynamic
functionality from within a script, rather than having it as a handler. Is
there a way to do this and still not have the cache problem?
2) What is the best way to install the libapreq objects, can the same
functionality be got through the Apache2 classes?
3) Point taken with the startup.pl, I will (even though there may be many)
'use' the packages in both places, I can see the benefit of sharing as much
memory as possible, thanks for the advice.
Kind regards,
Ben
-----Original Message-----
From: Philip M. Gollucci [mailto:pgollucci@p6m7g8.com]
Sent: 01 September 2006 06:58
To: ben.wilder@levexis.com
Cc: modperl@perl.apache.org
Subject: Re: Reloading .so files and .pm files
Ben Wilder wrote:
> Thanks Philip,
>
> Your advice is very helpful.
> you can guess whats coming next... running this under Mod_perl with 8
> 'StartServers', if i perform 8 requests with a particular $packageFileName
> and then attempt a different $packageFileName, Mod_perl will tell me that
> the package does not exist, because the 8 child processes all have a
> compiled version using the first $packageFileName that i specified and
> 'required'.
2.x example:
<Location /foo>
SetHandler modperl
PerlResponseHandler FOO
</Location>
package FOO;
#core
use strict;
use warnings FATAL => 'all'; no warnings qw(redefine);
use Carp;
#mp2
use Apache2::RequestRec ();
use Apache2::Const -compile => qw(OK);
use Apache2::Log ();
#libapreq
use APR::Request ();
use APR::Request::Apache2 ();
## Or this is already in PERL5INC some other way already
use lib qw(/path/to/lotsofpms);
sub handler {
my $r = shift;
my $req = APR::Request::Apache2->handle($r);
my $pkg = $req->param('pkg');
## VALIDATE $pkg VERY carefully for MALICOUS input!
## VALIDATE $pkg VERY carefully for MALICOUS input!
## VALIDATE $pkg VERY carefully for MALICOUS input!
## VALIDATE $pkg VERY carefully for MALICOUS input!
if (eval "require $pkg") {
## if this can fail or not exist, eval {} it
## or use UNIVERSAL::can()
$pkg->methodX(%args);
}
else {
$r->log_error("$pkg require failed\n")
}
return Apache2::Const::OK;
}
1;
Were it me, I would just have a 'use' statement for _ALL_ the files in your
startup.pl
and in this file.
BECAUSE if you serve a requests for X of the Y packages you use the memory
X, but its not shared with the 'require'
method. If you do the startup.pl/use thing, you still use the memory, but
its shared.
This also answers your question, even if the child compiles stuff it will
use the module you request everytime.
I'm guess you were using package/file scoped global vars before -- hence the
caching problem.
--
------------------------------------------------------------------------
Philip M. Gollucci (pgollucci@p6m7g8.com) 323.219.4708
Consultant / http://p6m7g8.net/Resume/resume.shtml
Senior Software Engineer - TicketMaster - http://ticketmaster.com
1024D/A79997FA F357 0FDD 2301 6296 690F 6A47 D55A 7172 A799 97F
"In all that I've done wrong I know I must have done something right to
deserve a hug every morning and butterfly kisses at night."
__ ___ ___ ____ __
/ |/ /_ __/ __/ __ \/ /
/ /|_/ / // /\ \/ /_/ / /__
/_/ /_/\_, /___/\___\_\___/
<___/
Re: Reloading .so files and .pm files
Posted by "Philip M. Gollucci" <pg...@p6m7g8.com>.
Ben Wilder wrote:
> Thanks Philip,
>
> Your advice is very helpful.
> you can guess whats coming next... running this under Mod_perl with 8
> 'StartServers', if i perform 8 requests with a particular $packageFileName
> and then attempt a different $packageFileName, Mod_perl will tell me that
> the package does not exist, because the 8 child processes all have a
> compiled version using the first $packageFileName that i specified and
> 'required'.
2.x example:
<Location /foo>
SetHandler modperl
PerlResponseHandler FOO
</Location>
package FOO;
#core
use strict;
use warnings FATAL => 'all'; no warnings qw(redefine);
use Carp;
#mp2
use Apache2::RequestRec ();
use Apache2::Const -compile => qw(OK);
use Apache2::Log ();
#libapreq
use APR::Request ();
use APR::Request::Apache2 ();
## Or this is already in PERL5INC some other way already
use lib qw(/path/to/lotsofpms);
sub handler {
my $r = shift;
my $req = APR::Request::Apache2->handle($r);
my $pkg = $req->param('pkg');
## VALIDATE $pkg VERY carefully for MALICOUS input!
## VALIDATE $pkg VERY carefully for MALICOUS input!
## VALIDATE $pkg VERY carefully for MALICOUS input!
## VALIDATE $pkg VERY carefully for MALICOUS input!
if (eval "require $pkg") {
## if this can fail or not exist, eval {} it
## or use UNIVERSAL::can()
$pkg->methodX(%args);
}
else {
$r->log_error("$pkg require failed\n")
}
return Apache2::Const::OK;
}
1;
Were it me, I would just have a 'use' statement for _ALL_ the files in your startup.pl
and in this file.
BECAUSE if you serve a requests for X of the Y packages you use the memory X, but its not shared with the 'require'
method. If you do the startup.pl/use thing, you still use the memory, but its shared.
This also answers your question, even if the child compiles stuff it will use the module you request everytime.
I'm guess you were using package/file scoped global vars before -- hence the caching problem.
--
------------------------------------------------------------------------
Philip M. Gollucci (pgollucci@p6m7g8.com) 323.219.4708
Consultant / http://p6m7g8.net/Resume/resume.shtml
Senior Software Engineer - TicketMaster - http://ticketmaster.com
1024D/A79997FA F357 0FDD 2301 6296 690F 6A47 D55A 7172 A799 97F
"In all that I've done wrong I know I must have done something right to
deserve a hug every morning and butterfly kisses at night."
__ ___ ___ ____ __
/ |/ /_ __/ __/ __ \/ /
/ /|_/ / // /\ \/ /_/ / /__
/_/ /_/\_, /___/\___\_\___/
<___/
RE: Reloading .so files and .pm files
Posted by Ben Wilder <be...@levexis.com>.
Thanks Philip,
Your advice is very helpful.
I wonder whether you could help with another question ?
As a follow up to my post earlier today (same scenario):
I have a package that performs the following:
It is used to dynamically require a particular .pm (from a directory of
100s) and call a generic method (common to all the .pms in the directory)
Which .pm package it requires is based on user input.
--Code snippet--
#Create package name and file name
my $packageName = $packageSchema . $packageId;
my $packageFileName = $packageName . ".pm";
#Attempt to 'require' the file
eval{require $packageFileName} or print "package does not exist";
#Attempt to execute the method call
my $execString = "$packageName" . "::" .
"GenericMethod('$methodArguments')";
my $resultString = eval $execString;
--Code snippet--
Each .pm file that is dynamically 'required' has a .so counterpart (the .pm
was generated by SWIG and is used to bootstrap the shared object)
you can guess whats coming next... running this under Mod_perl with 8
'StartServers', if i perform 8 requests with a particular $packageFileName
and then attempt a different $packageFileName, Mod_perl will tell me that
the package does not exist, because the 8 child processes all have a
compiled version using the first $packageFileName that i specified and
'required'.
I appreciate that this is exactly what Mod_perl is supposed to do, but has
anyone overcome this or any similar problems (conditionally 'requiring' code
in Mod_perl), and could you offer me any advice?
Once again, Very grateful for any advice you may have!
Regards
Ben
-----Original Message-----
From: Philip M. Gollucci [mailto:pgollucci@p6m7g8.com]
Sent: 31 August 2006 09:44
To: ben.wilder@levexis.com
Cc: modperl@perl.apache.org
Subject: Re: Reloading .so files and .pm files
Ben Wilder wrote:
> Question:
> What with the nature of the statistics I need to call from the compiled
> shared objects, I am recompiling the .so files about twice a day (as
> situations change). I need to update the .so files and their corresponding
> .pm module files.
>
> Does anyone have any suggestions as to how I could expire the old ones the
> Mod_perl processes would be using get Mod_perl to pick up the newly
updated
> .so and .pm files?
>
> I would have a directory of roughly 100 .so files and their corresponding
> .pm files (generated by SWIG) that would be dynamically 'required' by
> calling scripts, about half of these 100 shared objects would be
recompiled
> about twice a day!
>
> Any thoughts much appreciated!
It should be -- reloading the .pm file if it boostraps the XS component
normally should suffice
For 2.x:
PerlModule Apache2::Reload
PerlInitHandler Apache2::Reload
PerlSetVar ReloadAll Off
PerlSetVar ReloadModules "X::*"
#PerlSetVAr ReloadDebug On
X is the top-level package namespace for where your .pm files are.
If you have multiple directories that works too.
http://perl.apache.org/docs/2.0/user/coding/coding.html#Auto_Reloading_Modif
ied_Modules_with_Apache2__Reload
IF your builds are croned or even not, you could aways fire off a graceful
restart -- 2x a day isn't bad.
You're likely loosing a lot of memory "requiring" the modules at run-time
anyway, so its probably good you restart twice
a day -- unless you preload all of them at startup.
--
------------------------------------------------------------------------
Philip M. Gollucci (pgollucci@p6m7g8.com) 323.219.4708
Consultant / http://p6m7g8.net/Resume/resume.shtml
Senior Software Engineer - TicketMaster - http://ticketmaster.com
1024D/A79997FA F357 0FDD 2301 6296 690F 6A47 D55A 7172 A799 97F
"In all that I've done wrong I know I must have done something right to
deserve a hug every morning and butterfly kisses at night."
__ ___ ___ ____ __
/ |/ /_ __/ __/ __ \/ /
/ /|_/ / // /\ \/ /_/ / /__
/_/ /_/\_, /___/\___\_\___/
<___/
Re: Reloading .so files and .pm files
Posted by "Philip M. Gollucci" <pg...@p6m7g8.com>.
Ben Wilder wrote:
> Question:
> What with the nature of the statistics I need to call from the compiled
> shared objects, I am recompiling the .so files about twice a day (as
> situations change). I need to update the .so files and their corresponding
> .pm module files.
>
> Does anyone have any suggestions as to how I could expire the old ones the
> Mod_perl processes would be using get Mod_perl to pick up the newly updated
> .so and .pm files?
>
> I would have a directory of roughly 100 .so files and their corresponding
> .pm files (generated by SWIG) that would be dynamically 'required' by
> calling scripts, about half of these 100 shared objects would be recompiled
> about twice a day!
>
> Any thoughts much appreciated!
It should be -- reloading the .pm file if it boostraps the XS component normally should suffice
For 2.x:
PerlModule Apache2::Reload
PerlInitHandler Apache2::Reload
PerlSetVar ReloadAll Off
PerlSetVar ReloadModules "X::*"
#PerlSetVAr ReloadDebug On
X is the top-level package namespace for where your .pm files are.
If you have multiple directories that works too.
http://perl.apache.org/docs/2.0/user/coding/coding.html#Auto_Reloading_Modified_Modules_with_Apache2__Reload
IF your builds are croned or even not, you could aways fire off a graceful restart -- 2x a day isn't bad.
You're likely loosing a lot of memory "requiring" the modules at run-time anyway, so its probably good you restart twice
a day -- unless you preload all of them at startup.
--
------------------------------------------------------------------------
Philip M. Gollucci (pgollucci@p6m7g8.com) 323.219.4708
Consultant / http://p6m7g8.net/Resume/resume.shtml
Senior Software Engineer - TicketMaster - http://ticketmaster.com
1024D/A79997FA F357 0FDD 2301 6296 690F 6A47 D55A 7172 A799 97F
"In all that I've done wrong I know I must have done something right to
deserve a hug every morning and butterfly kisses at night."
__ ___ ___ ____ __
/ |/ /_ __/ __/ __ \/ /
/ /|_/ / // /\ \/ /_/ / /__
/_/ /_/\_, /___/\___\_\___/
<___/