You are viewing a plain text version of this content. The canonical link for it is here.
Posted to modperl@perl.apache.org by Stas Bekman <sb...@stason.org> on 2000/06/03 00:11:04 UTC
[benchmark] DBI/preload (was Re: [RFC] improving memory mapping thru code exercising)
On Mon, 29 May 2000, Vivek Khera wrote:
> >>>>> "SB" == Stas Bekman <sb...@stason.org> writes:
>
> SB> A while ago, a few people have mentioned that it's possible to improve the
> SB> way Perl data structures get mapped in memory pages, by exercising the
> SB> code before the child processes have been spawned, i.e. running the code
> SB> and not just pre-compiling it. Did anyone try this at home :) Any
> SB> satisfactionary results on this direction or what it just a crazy idea?
>
> This is extremely important for DBI, since the DBD layer is not loaded
> until needed. Thus, unless you exercise your Connect in DBI, the DBD
> is not loaded until each child is spawned. I'm sure other modules do
> some sort of initialization as well which could break sharing since
> perl code and data pages are the same as far as the OS is concerned.
In fact it's not the connect, but the install_driver that makes the
difference, connect_on_init (I know you said connect, not on_init :) only
gets the connection ready for the first request for each process, it
doesn't preload the driver!!! surprise, surprise... rush and update your
startup files!
Here are the tests I've conducted. Our goal is to find the startup
environment tha will lead to the smallest "Difference" between the shared
and normal memory reports, therefore lesser total memory usage.
Here are the results (exclusively for the mod_perl list ;-) :
The startup.pl file could have:
nothing added 1
install_driver 2
connect_on_init 3
Results (sorted by Diff):
=== First request:
Test: Size Shared Diff
---------------------------------------
2 : 3465216 2621440 843776 (bytes)
2+3 : 3461120 2609152 851968 (bytes)
1 : 3461120 2494464 966656 (bytes)
3 : 3461120 2482176 978944 (bytes)
=== Second request (all the subsequent request showed the same results):
Test: Size Shared Diff
---------------------------------------
2 : 3469312 2609152 860160 (bytes)
2+3 : 3481600 2605056 876544 (bytes)
1 : 3477504 2482176 995328 (bytes)
3 : 3481600 2469888 1011712 (bytes)
Notice that after the second reload the process's memory size grows a
little bit and there are less shared memory pages, but all the subsequent
reloads have shown the same results.
As you can see that the *real* winner is the startup.pl file that has used
install_driver (2). Having only connect_on_init (3) is the worst case in
terms of shared memory usage.
Here is the test enviroment:
### startup.pl:
* Always preloaded:
Gtop() and Apache::DBI()
* install_driver (2):
DBI->install_driver("mysql");
* connect_on_init (3):
Apache::DBI->connect_on_init('DBI:mysql:test::localhost',
"",
"",
{
PrintError => 1, # warn() on errors
RaiseError => 0, # don't die on error
AutoCommit => 1, # commit executes
# immediately
}
)
or DBI->disconnect("Cannot connect to database: $DBI::errstr\n");
### httpd.conf:
MinSpareServers 1
MaxSpareServers 1
StartServers 1
MaxClients 1
MaxRequestsPerChild 100
PerlModule Apache::Registry
### The registry script used in test (unmodified in all 4 test):
preload_dbi.pl
--------------
use strict;
use GTop ();
use DBI ();
my $dbh = DBI->connect("DBI:mysql:test::localhost",
"",
"",
{
PrintError => 1, # warn() on errors
RaiseError => 0, # don't die on error
AutoCommit => 1, # commit executes
# immediately
}
)
or DBI->disconnect("Cannot connect to database: $DBI::errstr\n");
my $r = shift;
$r->send_http_header('text/plain');
my $do_sql = "show tables";
my $sth = $dbh->prepare($do_sql);
$sth->execute();
my @data = ();
while (my @row = $sth->fetchrow_array){
push @data, @row;
}
$dbh->disconnect();
print "Data: @data\n";
my $proc_mem = GTop->new->proc_mem($$);
my $size = $proc_mem->size;
my $share = $proc_mem->share;
my $diff = $size - $share;
printf "%8s %8s %8s\n", qw(Size Shared Diff);
printf "%8d %8d %8d (bytes)\n",$size,$share,$diff;
---------------------
Your comments are very welcome!
Especially about other modules that allow initialization and thus
reducing the memory usage. (not CGI.pm, I know about this one)
_____________________________________________________________________
Stas Bekman JAm_pH -- Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide http://perl.apache.org/guide
mailto:stas@stason.org http://perl.org http://stason.org/TULARC
http://singlesheaven.com http://perlmonth.com http://sourcegarden.org
Re: [benchmark] DBI/preload (was Re: [RFC] improving memory mapping thru code exercising)
Posted by Stas Bekman <sb...@stason.org>.
On Mon, 5 Jun 2000, Vivek Khera wrote:
> >>>>> "PH" == Perrin Harkins <pe...@primenet.com> writes:
>
> PH> On Sat, 3 Jun 2000, Stas Bekman wrote:
> >> * install_driver (2):
> DBI-> install_driver("mysql");
>
> PH> I've never seen that before, and it isn't in the DBI perldoc. Is it safer
> PH> than "use DBD::mysql;"?
>
> "use DBD::mysql" doesn't really do anything, does it?
Only preloads (compiles) the module. See the benchmark.
_____________________________________________________________________
Stas Bekman JAm_pH -- Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide http://perl.apache.org/guide
mailto:stas@stason.org http://perl.org http://stason.org/TULARC
http://singlesheaven.com http://perlmonth.com http://sourcegarden.org
Re: [benchmark] DBI/preload (was Re: [RFC] improving memory mapping
thru code exercising)
Posted by Vivek Khera <kh...@kciLink.com>.
>>>>> "PH" == Perrin Harkins <pe...@primenet.com> writes:
PH> On Sat, 3 Jun 2000, Stas Bekman wrote:
>> * install_driver (2):
DBI-> install_driver("mysql");
PH> I've never seen that before, and it isn't in the DBI perldoc. Is it safer
PH> than "use DBD::mysql;"?
"use DBD::mysql" doesn't really do anything, does it?
Re: [benchmark] DBI/preload (was Re: [RFC] improving memory mapping thru code exercising)
Posted by Jason Terry <jt...@g-web.net>.
I just wanted to thank you guys for sending this to the mailing list.
I added these lines to my startup script....
use Carp;
CGI->compile(qw(my_common_functions));
DBI->install_driver('mysql');
Please note that these were already existing in my startup script.
use CGI();
use DBI();
And after testing, and running the server with the new settings all night long, it seems that I am saving ~2M of RAM for each apache
process. Again thank you.
However, this has made me curious, and left me wondering... I have 3 scripts that I pre-load in my startup file, would any of my
self-made modules, or other CPAN modules, benefit from including directly in the startup script... as opposed to being loaded only
in the pre-loaded scripts themselves?
Again, thank you for this thread it has saved me 20-80M of RAM depending on my current load.
-Jason
----- Original Message -----
From: "Stas Bekman" <sb...@stason.org>
To: "Tim Bunce" <Ti...@ig.co.uk>
Cc: "mod_perl list" <mo...@apache.org>
Sent: Sunday, June 04, 2000 1:50 PM
Subject: Re: [benchmark] DBI/preload (was Re: [RFC] improving memory mapping thru code exercising)
> On Sun, 4 Jun 2000, Tim Bunce wrote:
>
> > On Sat, Jun 03, 2000 at 02:49:47AM +0300, Stas Bekman wrote:
> > > On Fri, 2 Jun 2000, Perrin Harkins wrote:
> > >
> > > > On Sat, 3 Jun 2000, Stas Bekman wrote:
> > > > > * install_driver (2):
> > > > > DBI->install_driver("mysql");
> > > >
> > > > I've never seen that before,
> > >
> > > There is always a first time :)
> > >
> > > > and it isn't in the DBI perldoc.
> > >
> > > Where do you think I've found it :) It is mentioned in DBI manpage:
> > >
> > > DBI->connect automatically installs the driver if it
> > > has not been installed yet. Driver installation always
> > > returns a valid driver handle or it dies with an error
> > > message which includes the string 'install_driver' and
> > > the underlying problem. So, DBI->connect will die on a
> > > driver installation failure and will only return undef
> > > on a connect failure, for which $DBI::errstr will hold
> > > the error.
> >
> > I've been meaning to document it properly for the last couple of
> > years, since I knew people were using it for mod_perl.
>
> I guess most of the mod_perl folks don't use it, since they don't
> know about it. I have discovered it following the Vivek's reply.
>
> > > but actualy it's the DBD:: method.
> >
> > Nope. A DBI->method.
>
> Ok
>
> > > > Is it safer than "use DBD::mysql;"?
> > >
> > > "safer" is the wrong question. It's always used by DBI, but when you call
> > > it at the server startup you get a few more K shared, compared with only
> > > preloading of DBD::mysql.
> >
> > Yes, and the semantics of install_driver are that it'll die if the driver
> > can't be installed, just like 'use'.
> >
> > > Apache::DBI->connect_on_init('DBI:mysql:test::localhost', [...]
> > > or DBI->disconnect("Cannot connect to database: $DBI::errstr\n");
> >
> > There's no such method as DBI->disconnect.
>
> Ouch, you are right. But the first part has never failed for me, so I copy
> and paste this old error for years :) Thanks for telling me!
>
> _____________________________________________________________________
> Stas Bekman JAm_pH -- Just Another mod_perl Hacker
> http://stason.org/ mod_perl Guide http://perl.apache.org/guide
> mailto:stas@stason.org http://perl.org http://stason.org/TULARC
> http://singlesheaven.com http://perlmonth.com http://sourcegarden.org
Re: [benchmark] DBI/preload (was Re: [RFC] improving memory mapping thru code exercising)
Posted by Stas Bekman <sb...@stason.org>.
On Sun, 4 Jun 2000, Tim Bunce wrote:
> On Sat, Jun 03, 2000 at 02:49:47AM +0300, Stas Bekman wrote:
> > On Fri, 2 Jun 2000, Perrin Harkins wrote:
> >
> > > On Sat, 3 Jun 2000, Stas Bekman wrote:
> > > > * install_driver (2):
> > > > DBI->install_driver("mysql");
> > >
> > > I've never seen that before,
> >
> > There is always a first time :)
> >
> > > and it isn't in the DBI perldoc.
> >
> > Where do you think I've found it :) It is mentioned in DBI manpage:
> >
> > DBI->connect automatically installs the driver if it
> > has not been installed yet. Driver installation always
> > returns a valid driver handle or it dies with an error
> > message which includes the string 'install_driver' and
> > the underlying problem. So, DBI->connect will die on a
> > driver installation failure and will only return undef
> > on a connect failure, for which $DBI::errstr will hold
> > the error.
>
> I've been meaning to document it properly for the last couple of
> years, since I knew people were using it for mod_perl.
I guess most of the mod_perl folks don't use it, since they don't
know about it. I have discovered it following the Vivek's reply.
> > but actualy it's the DBD:: method.
>
> Nope. A DBI->method.
Ok
> > > Is it safer than "use DBD::mysql;"?
> >
> > "safer" is the wrong question. It's always used by DBI, but when you call
> > it at the server startup you get a few more K shared, compared with only
> > preloading of DBD::mysql.
>
> Yes, and the semantics of install_driver are that it'll die if the driver
> can't be installed, just like 'use'.
>
> > Apache::DBI->connect_on_init('DBI:mysql:test::localhost', [...]
> > or DBI->disconnect("Cannot connect to database: $DBI::errstr\n");
>
> There's no such method as DBI->disconnect.
Ouch, you are right. But the first part has never failed for me, so I copy
and paste this old error for years :) Thanks for telling me!
_____________________________________________________________________
Stas Bekman JAm_pH -- Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide http://perl.apache.org/guide
mailto:stas@stason.org http://perl.org http://stason.org/TULARC
http://singlesheaven.com http://perlmonth.com http://sourcegarden.org
Re: [benchmark] DBI/preload (was Re: [RFC] improving memory mapping
thru code exercising)
Posted by Tim Bunce <Ti...@ig.co.uk>.
On Sat, Jun 03, 2000 at 02:49:47AM +0300, Stas Bekman wrote:
> On Fri, 2 Jun 2000, Perrin Harkins wrote:
>
> > On Sat, 3 Jun 2000, Stas Bekman wrote:
> > > * install_driver (2):
> > > DBI->install_driver("mysql");
> >
> > I've never seen that before,
>
> There is always a first time :)
>
> > and it isn't in the DBI perldoc.
>
> Where do you think I've found it :) It is mentioned in DBI manpage:
>
> DBI->connect automatically installs the driver if it
> has not been installed yet. Driver installation always
> returns a valid driver handle or it dies with an error
> message which includes the string 'install_driver' and
> the underlying problem. So, DBI->connect will die on a
> driver installation failure and will only return undef
> on a connect failure, for which $DBI::errstr will hold
> the error.
I've been meaning to document it properly for the last couple of
years, since I knew people were using it for mod_perl.
> but actualy it's the DBD:: method.
Nope. A DBI->method.
> > Is it safer than "use DBD::mysql;"?
>
> "safer" is the wrong question. It's always used by DBI, but when you call
> it at the server startup you get a few more K shared, compared with only
> preloading of DBD::mysql.
Yes, and the semantics of install_driver are that it'll die if the driver
can't be installed, just like 'use'.
> Apache::DBI->connect_on_init('DBI:mysql:test::localhost', [...]
> or DBI->disconnect("Cannot connect to database: $DBI::errstr\n");
There's no such method as DBI->disconnect.
Tim.
Re: [benchmark] DBI/preload (was Re: [RFC] improving memory mappingthru
code exercising)
Posted by Ed Phillips <ed...@homewarehouse.com>.
Yes, very cool Stas!
Perrin Harkins wrote:
> On Sat, 3 Jun 2000, Stas Bekman wrote:
>
> > correction for the 3rd version (had the wrong startup), but it's almost
> > the same.
> >
> > Version Size Shared Diff Test type
> > --------------------------------------------------------------------
> > 1 3469312 2609152 860160 install_driver
> > 2 3481600 2605056 876544 install_driver & connect_on_init
> > 3 3469312 2588672 880640 preload driver
> > 4 3477504 2482176 995328 nothing added
> > 5 3481600 2469888 1011712 connect_on_init
>
> Cool, thanks for running the test! I will put this information to good
> use...
Re: [benchmark] DBI/preload (was Re: [RFC] improving memory mapping
thru code exercising)
Posted by Perrin Harkins <pe...@primenet.com>.
On Sat, 3 Jun 2000, Stas Bekman wrote:
> correction for the 3rd version (had the wrong startup), but it's almost
> the same.
>
> Version Size Shared Diff Test type
> --------------------------------------------------------------------
> 1 3469312 2609152 860160 install_driver
> 2 3481600 2605056 876544 install_driver & connect_on_init
> 3 3469312 2588672 880640 preload driver
> 4 3477504 2482176 995328 nothing added
> 5 3481600 2469888 1011712 connect_on_init
Cool, thanks for running the test! I will put this information to good
use...
Re: [benchmark] DBI/preload (was Re: [RFC] improving memory mapping thru code exercising)
Posted by Stas Bekman <sb...@stason.org>.
> but actualy it's the DBD:: method. No matter what, you get more shared
> memory with it, see the updated table:
>
> Version Size Shared Diff Test type
> --------------------------------------------------------------------
> 1 3469312 2609152 860160 install_driver
> 2 3481600 2605056 876544 install_driver & connect_on_init
> 3 3469312 2576384 892928 preload driver
> 4 3477504 2482176 995328 nothing added
> 5 3481600 2469888 1011712 connect_on_init
correction for the 3rd version (had the wrong startup), but it's almost
the same.
Version Size Shared Diff Test type
--------------------------------------------------------------------
1 3469312 2609152 860160 install_driver
2 3481600 2605056 876544 install_driver & connect_on_init
3 3469312 2588672 880640 preload driver
4 3477504 2482176 995328 nothing added
5 3481600 2469888 1011712 connect_on_init
_____________________________________________________________________
Stas Bekman JAm_pH -- Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide http://perl.apache.org/guide
mailto:stas@stason.org http://perl.org http://stason.org/TULARC
http://singlesheaven.com http://perlmonth.com http://sourcegarden.org
Re: [benchmark] DBI/preload (was Re: [RFC] improving memory mapping thru code exercising)
Posted by Stas Bekman <sb...@stason.org>.
On Fri, 2 Jun 2000, Perrin Harkins wrote:
> On Sat, 3 Jun 2000, Stas Bekman wrote:
> > * install_driver (2):
> > DBI->install_driver("mysql");
>
> I've never seen that before,
There is always a first time :)
> and it isn't in the DBI perldoc.
Where do you think I've found it :) It is mentioned in DBI manpage:
DBI->connect automatically installs the driver if it
has not been installed yet. Driver installation always
returns a valid driver handle or it dies with an error
message which includes the string 'install_driver' and
the underlying problem. So, DBI->connect will die on a
driver installation failure and will only return undef
on a connect failure, for which $DBI::errstr will hold
the error.
but actualy it's the DBD:: method. No matter what, you get more shared
memory with it, see the updated table:
Version Size Shared Diff Test type
--------------------------------------------------------------------
1 3469312 2609152 860160 install_driver
2 3481600 2605056 876544 install_driver & connect_on_init
3 3469312 2576384 892928 load driver
4 3477504 2482176 995328 nothing added
5 3481600 2469888 1011712 connect_on_init
> Is it safer than "use DBD::mysql;"?
"safer" is the wrong question. It's always used by DBI, but when you call
it at the server startup you get a few more K shared, compared with only
preloading of DBD::mysql.
_____________________________________________________________________
Stas Bekman JAm_pH -- Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide http://perl.apache.org/guide
mailto:stas@stason.org http://perl.org http://stason.org/TULARC
http://singlesheaven.com http://perlmonth.com http://sourcegarden.org
Re: [benchmark] DBI/preload (was Re: [RFC] improving memory mapping
thru code exercising)
Posted by Perrin Harkins <pe...@primenet.com>.
On Sat, 3 Jun 2000, Stas Bekman wrote:
> * install_driver (2):
> DBI->install_driver("mysql");
I've never seen that before, and it isn't in the DBI perldoc. Is it safer
than "use DBD::mysql;"?
- Perrin