You are viewing a plain text version of this content. The canonical link for it is here.
Posted to embperl@perl.apache.org by Neil Gunton <ne...@home.nilspace.com> on 2002/05/01 21:00:14 UTC
Preloading modules to increase shared memory
Hi Gerald,
I am trying to improve the shared memory usage on my server, and I've managed to
(apparently) preload all the Embperl files on two of my sites. It all seems to
work, at first. But then the shared memory goes down quite rapidly. On initial
startup, Apache::VMonitor says that most of the memory is shared, but after a
couple of hundred requests (per child) it has fallen to around half of the total
memory taken by each process. A typical total size is 33.9M, with 17.7M shared
after 250 requests (for that child). I realize that Perl is probably responsible
for this, and I have read the mod_perl performance document, but I just wanted
to check in and make sure I am doing the right sort of thing here...
First of all, I had to change my EmbperlObject files to enable preloading (using
1.3.4). The problem lay in lines like this, which are at the start of every
subroutine module:
[! Execute ({isa => '../Journal.epl'}) !]
I found that I had to change this to an absolute path, like this:
[! Execute ({isa => '/full/path/to/parent/Journal.epl'}) !]
Is this because EmbperlObject uses the filename given in Execute as a key to
some kind of internal hash?
I used Embperl::Execute in startup.epl (see below). I also had to remove the
'package => __PACKAGE__' from the files which are executed from base.epl, which
meant putting all the (formerly global) variables in the $req variable which is
passed in to every module. I experimented with using the global:: namespace, but
this seemed to cause some problems with preloaded modules: Everything would seem
to work for a while, but then there would be spurious errors along the lines of
'attempt to modify read-only value' when the variable was being initialized. So,
I got rid of all global:: vars and now it all seems to work.
I wrote a routine in startup.pl to recursively traverse the directory tree and
preload all the Embperl files which match a given pattern. Even though this is
an EmbperlObject website, I found that calling Embperl::Execute seemed to work
fine. Is that ok? Calling EmbperlObject::Execute gave these messages for every
file:
Use of uninitialized value in substitution (s///) at
/usr/lib/perl5/site_perl/5.6.0/i386-linux/HTML/EmbperlObject.pm line 181.
Use of uninitialized value in bitwise and (&) at
/usr/lib/perl5/site_perl/5.6.0/i386-linux/HTML/EmbperlObject.pm line 187.
Use of uninitialized value in concatenation (.) at
/usr/lib/perl5/site_perl/5.6.0/i386-linux/HTML/EmbperlObject.pm line 213.
I am just wondering if I am doing the right thing here, since after a while the
shared memory seems to drift down to pretty much what it was before I started
preloading the Embperl files. So is it really worth it? Here's the startup.pl,
I'd appreciate any advice as to whether I am doing something really braindead
here...
Thanks in advance,
-Neil
startup.pl:
#!/usr/local/bin/perl
# First modify the include path
BEGIN
{
use Apache ();
use lib '/www/lib/perl';
}
# Common modules
use Apache::Registry ();
use Apache::Constants ();
use Apache::File ();
use Apache::Log ();
use Safe ();
use Log::Logger ();
use File::Copy ();
use File::Path ();
use File::Glob ();
use Time::Zone ();
use CGI qw (-compile :cookie cgi_error header);
use Date::Calc qw(:all);
use Image::Magick ();
use HTML::Embperl ();
use HTML::EmbperlObject ();
use DBI ();
DBI->install_driver('mysql');
# My modules
use Apache::BlockAgent ();
use Apache::Nilspace::Access ();
use Apache::Nilspace::Handler ();
use Nilspace ();
use Nilspace::Agenda ();
use Nilspace::Commerce ();
use Nilspace::Mail ();
# Apache::VMonitor
use Apache::VMonitor();
$Apache::VMonitor::Config{BLINKING} = 1;
$Apache::VMonitor::Config{REFRESH} = 0;
$Apache::VMonitor::Config{VERBOSE} = 0;
$Apache::VMonitor::Config{SYSTEM} = 1;
$Apache::VMonitor::Config{APACHE} = 1;
$Apache::VMonitor::Config{PROCS} = 1;
$Apache::VMonitor::Config{MOUNT} = 1;
$Apache::VMonitor::Config{FS_USAGE} = 1;
$Apache::VMonitor::Config{SORT_BY} = 'size';
$Apache::VMonitor::PROC_REGEX = join "\|", qw(httpd mysql squid);
# Preload Embperl website code
preload_dir ('/www/lib/perl/Apache', '*.html *.epl');
preload_dir ('/www/neilgunton.com/htdocs', '*.html *.epl');
preload_dir ('/www/crazyguyonabike.com/htdocs', '*.html *.epl');
# Recursive directory traversal sub which preloads Embperl files
sub preload_dir
{
my ($dir, # The current directory which is to be processed
$pattern # A pattern identifying files to be processed
) = @_;
local *DIR;
opendir (DIR, $dir) or die "Could not open directory: $dir: $!";
# First, process files in this directory
# Pattern consists of a potential list of patterns,
# separated by spaces.
# First we make a list of patterns, and then glob each of these
foreach my $glob (split (/\s/, $pattern))
{
# Iterate through the resulting list of files
foreach my $file (File::Glob::glob ("$dir/$glob"))
{
if (!(-d $file) && (-e $file))
{
print "Embperl::Execute $file\n";
HTML::EmbperlObject::Execute ({inputfile => $file,
import => 0,
escmode => 0,
options => 16}) ;
}
}
}
# Now, recursively go down into subdirectories
while (defined(my $file = readdir (DIR)))
{
# Only recurse on directories, which do not start with ".",
# and skip symbolic links
if (-d "$dir/$file" &&
!(-l "$dir/$file") &&
($file !~ /^\.{1,2}$/))
{
preload_dir ("$dir/$file", $pattern);
}
}
}
1;
---------------------------------------------------------------------
To unsubscribe, e-mail: embperl-unsubscribe@perl.apache.org
For additional commands, e-mail: embperl-help@perl.apache.org
Re: Preloading modules to increase shared memory
Posted by Gerald Richter <ri...@ecos.de>.
Hi Neil,
I think what you are doing is correct. I guess the problem is that Perl is
storing it's compiled code and data in the same memory pages, so when the
data gets modified, the code isn't also shared anymore. Embperl 2.0 may
behave better, because it compiled all the code as whole chunck, so it's
less likely that code and data gets mixed up in the same memory page.
Maybe you gain a little memory by limitied the requests per childs (e.g. to
100). Since you already have compiled the perl code in the parent, a child
startup should not cost to much.
>
> First of all, I had to change my EmbperlObject files to enable preloading
(using
> 1.3.4). The problem lay in lines like this, which are at the start of
every
> subroutine module:
>
> [! Execute ({isa => '../Journal.epl'}) !]
>
> I found that I had to change this to an absolute path, like this:
>
> [! Execute ({isa => '/full/path/to/parent/Journal.epl'}) !]
>
> Is this because EmbperlObject uses the filename given in Execute as a key
to
> some kind of internal hash?
>
Embperl object sets up a search path. You can do the same for your
preloading with the path parameter.
> I
> wonder if Apache 2.x will handle this any better with threads. No idea how
all
> that works on Linux...
Apache/mod_perl 2.0 use Perl's ithreads, there Perl will take care to
separate data and code and Perl will make sure to share the code among all
threads. I expected it to be a great inprovement in memory uitlization
Gerald
-------------------------------------------------------------
Gerald Richter ecos electronic communication services gmbh
Internetconnect * Webserver/-design/-datenbanken * Consulting
Post: Tulpenstrasse 5 D-55276 Dienheim b. Mainz
E-Mail: richter@ecos.de Voice: +49 6133 925131
WWW: http://www.ecos.de Fax: +49 6133 925152
-------------------------------------------------------------
---------------------------------------------------------------------
To unsubscribe, e-mail: embperl-unsubscribe@perl.apache.org
For additional commands, e-mail: embperl-help@perl.apache.org
Re: Preloading modules to increase shared memory
Posted by Ed Grimm <ed...@asgard.rsc.raytheon.com>.
How is your swap space? Linux doesn't count swapped shared memory as
shared, only shared real memory. (Incidentally, Solaris and a number of
other unix OSes don't have swapped shared memory. As I seem to be one
of the minority running Embperl on non-Linux, few seem to care about
that aspect of things.)
Ed
On Wed, 1 May 2002, Neil Gunton wrote:
> Hi Gerald,
>
> I am trying to improve the shared memory usage on my server, and I've managed to
> (apparently) preload all the Embperl files on two of my sites. It all seems to
> work, at first. But then the shared memory goes down quite rapidly. On initial
> startup, Apache::VMonitor says that most of the memory is shared, but after a
> couple of hundred requests (per child) it has fallen to around half of the total
> memory taken by each process. A typical total size is 33.9M, with 17.7M shared
> after 250 requests (for that child). I realize that Perl is probably responsible
> for this, and I have read the mod_perl performance document, but I just wanted
> to check in and make sure I am doing the right sort of thing here...
>
> First of all, I had to change my EmbperlObject files to enable preloading (using
> 1.3.4). The problem lay in lines like this, which are at the start of every
> subroutine module:
>
> [! Execute ({isa => '../Journal.epl'}) !]
>
> I found that I had to change this to an absolute path, like this:
>
> [! Execute ({isa => '/full/path/to/parent/Journal.epl'}) !]
>
> Is this because EmbperlObject uses the filename given in Execute as a key to
> some kind of internal hash?
>
> I used Embperl::Execute in startup.epl (see below). I also had to remove the
> 'package => __PACKAGE__' from the files which are executed from base.epl, which
> meant putting all the (formerly global) variables in the $req variable which is
> passed in to every module. I experimented with using the global:: namespace, but
> this seemed to cause some problems with preloaded modules: Everything would seem
> to work for a while, but then there would be spurious errors along the lines of
> 'attempt to modify read-only value' when the variable was being initialized. So,
> I got rid of all global:: vars and now it all seems to work.
>
> I wrote a routine in startup.pl to recursively traverse the directory tree and
> preload all the Embperl files which match a given pattern. Even though this is
> an EmbperlObject website, I found that calling Embperl::Execute seemed to work
> fine. Is that ok? Calling EmbperlObject::Execute gave these messages for every
> file:
>
> Use of uninitialized value in substitution (s///) at
> /usr/lib/perl5/site_perl/5.6.0/i386-linux/HTML/EmbperlObject.pm line 181.
> Use of uninitialized value in bitwise and (&) at
> /usr/lib/perl5/site_perl/5.6.0/i386-linux/HTML/EmbperlObject.pm line 187.
> Use of uninitialized value in concatenation (.) at
> /usr/lib/perl5/site_perl/5.6.0/i386-linux/HTML/EmbperlObject.pm line 213.
>
> I am just wondering if I am doing the right thing here, since after a while the
> shared memory seems to drift down to pretty much what it was before I started
> preloading the Embperl files. So is it really worth it? Here's the startup.pl,
> I'd appreciate any advice as to whether I am doing something really braindead
> here...
>
> Thanks in advance,
>
> -Neil
>
> startup.pl:
> #!/usr/local/bin/perl
>
> # First modify the include path
> BEGIN
> {
> use Apache ();
> use lib '/www/lib/perl';
> }
>
> # Common modules
> use Apache::Registry ();
> use Apache::Constants ();
> use Apache::File ();
> use Apache::Log ();
> use Safe ();
> use Log::Logger ();
> use File::Copy ();
> use File::Path ();
> use File::Glob ();
> use Time::Zone ();
> use CGI qw (-compile :cookie cgi_error header);
> use Date::Calc qw(:all);
> use Image::Magick ();
> use HTML::Embperl ();
> use HTML::EmbperlObject ();
> use DBI ();
> DBI->install_driver('mysql');
>
> # My modules
> use Apache::BlockAgent ();
> use Apache::Nilspace::Access ();
> use Apache::Nilspace::Handler ();
> use Nilspace ();
> use Nilspace::Agenda ();
> use Nilspace::Commerce ();
> use Nilspace::Mail ();
>
> # Apache::VMonitor
> use Apache::VMonitor();
> $Apache::VMonitor::Config{BLINKING} = 1;
> $Apache::VMonitor::Config{REFRESH} = 0;
> $Apache::VMonitor::Config{VERBOSE} = 0;
> $Apache::VMonitor::Config{SYSTEM} = 1;
> $Apache::VMonitor::Config{APACHE} = 1;
> $Apache::VMonitor::Config{PROCS} = 1;
> $Apache::VMonitor::Config{MOUNT} = 1;
> $Apache::VMonitor::Config{FS_USAGE} = 1;
> $Apache::VMonitor::Config{SORT_BY} = 'size';
> $Apache::VMonitor::PROC_REGEX = join "\|", qw(httpd mysql squid);
>
> # Preload Embperl website code
> preload_dir ('/www/lib/perl/Apache', '*.html *.epl');
> preload_dir ('/www/neilgunton.com/htdocs', '*.html *.epl');
> preload_dir ('/www/crazyguyonabike.com/htdocs', '*.html *.epl');
>
> # Recursive directory traversal sub which preloads Embperl files
> sub preload_dir
> {
> my ($dir, # The current directory which is to be processed
> $pattern # A pattern identifying files to be processed
> ) = @_;
>
> local *DIR;
> opendir (DIR, $dir) or die "Could not open directory: $dir: $!";
>
> # First, process files in this directory
> # Pattern consists of a potential list of patterns,
> # separated by spaces.
> # First we make a list of patterns, and then glob each of these
> foreach my $glob (split (/\s/, $pattern))
> {
> # Iterate through the resulting list of files
> foreach my $file (File::Glob::glob ("$dir/$glob"))
> {
> if (!(-d $file) && (-e $file))
> {
> print "Embperl::Execute $file\n";
> HTML::EmbperlObject::Execute ({inputfile => $file,
> import => 0,
> escmode => 0,
> options => 16}) ;
> }
> }
> }
>
> # Now, recursively go down into subdirectories
> while (defined(my $file = readdir (DIR)))
> {
> # Only recurse on directories, which do not start with ".",
> # and skip symbolic links
> if (-d "$dir/$file" &&
> !(-l "$dir/$file") &&
> ($file !~ /^\.{1,2}$/))
> {
> preload_dir ("$dir/$file", $pattern);
> }
> }
> }
>
> 1;
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: embperl-unsubscribe@perl.apache.org
> For additional commands, e-mail: embperl-help@perl.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: embperl-unsubscribe@perl.apache.org
For additional commands, e-mail: embperl-help@perl.apache.org