You are viewing a plain text version of this content. The canonical link for it is here.
Posted to modperl@perl.apache.org by Badai Aqrandista <ba...@hotmail.com> on 2005/08/15 06:23:56 UTC
web application speed problem with OO modules
Hi all,
I'm new to the list and I have a question related to mod_perl... I have
tried my best to solve the problem, but failed... So I turn to you guys...
I'm currently in the process of optimizing a my company's web application
that I've rewritten almost from scratch for 4 months. The older version uses
Apache::Registry, slightly modified cgi scripts and procedural perl modules.
Whereas the new version is a rewrite of all those into object oriented
modules, with Template Toolkit as the templating system. The code has become
much more manageable now.
However, the speed of the new code is a LOT worse than what I had expected.
I have optimized the code with caches, indexes and rewriting inefficient SQL
queries. But the speed is still 2 to 4 times slower than the old version for
the same algorithm. (Of course I only optimized what the profiler tells me).
(I only try to improve the speed of the code, so the use of proxy shouldn't
be relevant here - I think, I could be wrong).
Does anyone has any other suggestion to improve a web application's speed?
I'd give more details if needed...
Thanks for your attentions...
---
Badai Aqrandista
Cheepy (?)
_________________________________________________________________
Dating? Try Lavalife get 7 days FREE! Sign up NOW.
http://www.lavalife.com/clickthru/clickthru.act?id=ninemsn&context=an99&a=20233&locale=en_AU&_t=33473
Re: web application speed problem with OO modules
Posted by Frank Wiles <fr...@wiles.org>.
On Mon, 15 Aug 2005 11:42:52 -0400
Perrin Harkins <pe...@elem.com> wrote:
> Frank Wiles wrote:
> > Give Apache::SmallProf a try. Where Apache::DProf shows you the
> > subroutine calls, SmallProf will show you a profile on a line by
> > line basis. This should help you zero in on what is causing the
> > slowness.
>
> I'd say just the opposite in most cases. Usually you want to see
> which subs are taking the time. SmallProf is mostly useful when your
> subs are much too big.
>
> It is very common though for people to not properly initialize the
> debugger when using DProf and thus not get results for most of their
> code. That often leads them to think they should use a different
> profiler, like SmallProf.
Yup that's exactly what I was thinking, I should have thought about
the initialization issue.
---------------------------------
Frank Wiles <fr...@wiles.org>
http://www.wiles.org
---------------------------------
Re: web application speed problem with OO modules
Posted by Perrin Harkins <pe...@elem.com>.
Frank Wiles wrote:
> Give Apache::SmallProf a try. Where Apache::DProf shows you the
> subroutine calls, SmallProf will show you a profile on a line by line
> basis. This should help you zero in on what is causing the slowness.
I'd say just the opposite in most cases. Usually you want to see which
subs are taking the time. SmallProf is mostly useful when your subs are
much too big.
It is very common though for people to not properly initialize the
debugger when using DProf and thus not get results for most of their
code. That often leads them to think they should use a different
profiler, like SmallProf.
- Perrin
Re: web application speed problem with OO modules
Posted by Frank Wiles <fr...@wiles.org>.
On Mon, 15 Aug 2005 14:59:00 +1000
"Badai Aqrandista" <ba...@hotmail.com> wrote:
> I doubt the ApacheHandler::handler subroutine runs that long, because
> it only shows the accumulated time used by other subroutines it
> calls. That makes the profile isn't really accurate. How do I make it
> to show the statistics of the subroutines called from the handler?
Give Apache::SmallProf a try. Where Apache::DProf shows you the
subroutine calls, SmallProf will show you a profile on a line by line
basis. This should help you zero in on what is causing the slowness.
---------------------------------
Frank Wiles <fr...@wiles.org>
http://www.wiles.org
---------------------------------
Re: apache children waits for each other?
Posted by David Hodgkinson <da...@hodgkinson.org>.
On 16 Aug 2005, at 06:55, Badai Aqrandista wrote:
>
>
>>
>> Badai Aqrandista wrote:
>>
>>> My mod_perl web app uses memcached to cache most of the (MySQL)
>>> database query results and as the session storage
>>> (Apache::Session::Memcached).
>>>
>>
>> Would it be a problem for your application if you suddenly lost
>> all of your session data? That could happen with memcached.
>>
>
> Yes, I'm aware of that. The problem is that the database structure
> is a reminiscent of the old version, which creates one database per
> client. It used to be only 40 clients and one web server. Now we
> have 300+ clients (=300+ databases) and 2 web servers. I always get
> 'Too many connections' error when I test the application with more
> than 100 concurrent request. Any other ideas?
Yes, really read an understand the chapters in the mod_perl guide about
setting up two_tier servers and what the implications are.
You may need to do some MySQL tuning too if you haven't already.
There are
parameters in there to do with the number of concurrent open databases.
Re: apache children waits for each other?
Posted by Badai Aqrandista <ba...@hotmail.com>.
> > I don't know it's even possible in mysql. Its reference docs doesn't say
> > that it is (http://dev.mysql.com/doc/mysql/en/join.html).
> >
>
>Yes it does :
>http://dev.mysql.com/doc/mysql/en/identifier-qualifiers.html
>
Hmmm... Excellent!!! I'll use it then...Will let you know how it goes...
Thanks, mate...
---
Badai Aqrandista
Cheepy (?)
_________________________________________________________________
Single? Start dating at Lavalife. Try our 7 day FREE trial!
http://lavalife9.ninemsn.com.au/clickthru/clickthru.act?context=an99&locale=en_AU&a=19179
Re: apache children waits for each other?
Posted by Clinton Gormley <cl...@traveljury.com>.
> I don't know it's even possible in mysql. Its reference docs doesn't say
> that it is (http://dev.mysql.com/doc/mysql/en/join.html).
>
Yes it does :
http://dev.mysql.com/doc/mysql/en/identifier-qualifiers.html
________________________________________________________________________
Clinton Gormley clinton@traveljury.com
www.TravelJury.com - For travellers, By travellers
Re: apache children waits for each other?
Posted by Badai Aqrandista <ba...@hotmail.com>.
Hi,
>I'm assuming that each client database is contained within the same
>MySQL server, and that you're not running 300 MySQL servers on different
>machines or different ports?
Yes, why would I do that?
>If so, you can reuse the same connections, and just reference the
>different database in the query.So :
>
>select *
>from db1.table1
>
>select *
>from db2.table1
>
>When you connect and specify a database name, you're just specifying the
>default database.
I don't know it's even possible in mysql. Its reference docs doesn't say
that it is (http://dev.mysql.com/doc/mysql/en/join.html).
Thanks...
---
Badai Aqrandista
Cheepy (?)
_________________________________________________________________
REALESTATE: biggest buy/rent/share listings
http://ninemsn.realestate.com.au
Re: apache children waits for each other?
Posted by Clinton Gormley <cl...@traveljury.com>.
> Yes, I'm aware of that. The problem is that the database structure is a
> reminiscent of the old version, which creates one database per client. It
> used to be only 40 clients and one web server. Now we have 300+ clients
> (=300+ databases) and 2 web servers. I always get 'Too many connections'
> error when I test the application with more than 100 concurrent request. Any
> other ideas?
>
I'm assuming that each client database is contained within the same
MySQL server, and that you're not running 300 MySQL servers on different
machines or different ports?
If so, you can reuse the same connections, and just reference the
different database in the query.So :
select *
from db1.table1
select *
from db2.table1
When you connect and specify a database name, you're just specifying the
default database.
________________________________________________________________________
Clinton Gormley clinton@traveljury.com
www.TravelJury.com - For travellers, By travellers
Re: apache children waits for each other?
Posted by Jonathan Vanasco <jv...@mastersofbranding.com>.
On Aug 16, 2005, at 11:17 AM, Perrin Harkins wrote:
> Good plan. This would make a nice addition (as a separate module) to
> the Apache::Session::Memcached distribution.
I'll see if I can figure out how to make:
Apache::Session::Memcached::WithFailover
where it does that behavior, but then allows you to specify a specific
Apache::Session datastore as being the failover device
it shouldn't be all that hard - mostly aliasing function calls to
existing modules.
Re: apache children waits for each other?
Posted by Perrin Harkins <pe...@elem.com>.
Jonathan Vanasco wrote:
> I do all of my memcached stuff , including sessions, with mysql
> failover.
Good plan. This would make a nice addition (as a separate module) to
the Apache::Session::Memcached distribution.
- Perrin
Re: apache children waits for each other?
Posted by Badai Aqrandista <ba...@hotmail.com>.
>I do all of my memcached stuff , including sessions, with mysql failover.
>its barely more code - i just make every public function address two
>private functions.
>
>ie:
>
>sub save {
> $_[0]->_save_memcached();
> $_[0]->_save_mysql();
>}
>
>sub load {
> if ( !$_[0]->_load_memcached() )
> {
> $_[0]->_load_mysql();
> }
>}
>
I thought about the same thing last night, but is there any race condition
problem if I use it for session?
---
Badai Aqrandista
Cheepy (?)
_________________________________________________________________
SEEK: Over 80,000 jobs across all industries at Australia's #1 job site.
http://ninemsn.seek.com.au?hotmail
Re: apache children waits for each other?
Posted by Jonathan Vanasco <jv...@mastersofbranding.com>.
On Aug 16, 2005, at 9:49 AM, Perrin Harkins wrote:
> If you can use Apache::Session::Memcached, why can't you use
> Apache::Session::MySQL with a common database login? That will mean
> one connection per apache process. You can also raise the connection
> limit on your database server if it isn't dying under the load.
I do all of my memcached stuff , including sessions, with mysql
failover. its barely more code - i just make every public function
address two private functions.
ie:
sub save {
$_[0]->_save_memcached();
$_[0]->_save_mysql();
}
sub load {
if ( !$_[0]->_load_memcached() )
{
$_[0]->_load_mysql();
}
}
Re: apache children waits for each other?
Posted by Badai Aqrandista <ba...@hotmail.com>.
>Apache::DProf uses the debugger hooks. You can also try Sam Tregar's
>Devel::Profiler::Apache which is somewhat slower but easier to use since it
>doesn't use the debugger.
I'll look into that...
Thanks for your suggestions...
---
Badai Aqrandista
Cheepy (?)
_________________________________________________________________
REALESTATE: biggest buy/rent/share listings
http://ninemsn.realestate.com.au
Re: apache children waits for each other?
Posted by Perrin Harkins <pe...@elem.com>.
Badai Aqrandista wrote:
> <Perl>
> use lib '/i4u/web/elres-mp';
> # require Apache::DB;
> # Apache::DB->init;
> </Perl>
> PerlRequire /i4u/web/elres-mp/etc/startup.pl
>
> PerlModule Apache::DProf
>
> PerlModule ELRes::ELRes
> PerlModule ELRes::ApacheHandler
> PerlTransHandler ELRes::ApacheHandler->trans_handler
> PerlLogHandler ELRes::ApacheHandler->log_handler
> PerlCleanupHandler ELRes::ApacheHandler->cleanup_handler
This looks okay, assuming those lines were not commented out when you
ran it.
> the startup.pl script loads and require's all modules under ELRes::
> namespace on startup time, as suggested in mod_perl documentation...
Why do you have those two PerlModule calls then? I suggest taking them
out. If that breaks anything, it means your startup.pl is not doing
what you think it is.
> the object that $app refer to is dynamically decided based on the
> hostname, for example:
>
> a.example.com ---> $app = ELRes::Property->new()
> b.example.com ---> $app = ELRes::Property->new()
> c.example.com ---> $app = ELRes::Distributor->new()
>
> both ELRes::Property and ELRes::Distributor are children of
> ELRes::TopLevelEntity and have run() method...
>
> does this prevents the debugger to know which subroutine should be
> profiled?
No, it won't make a difference.
> Hmmm... Where can I learn more about the debugger?
There is information about it in the perl man pages (man perldebug), in
Programming Perl, and in various articles on the web.
> I didn't know profiling needs debugger, not until I was told about it in
> this mailinglist...
Apache::DProf uses the debugger hooks. You can also try Sam Tregar's
Devel::Profiler::Apache which is somewhat slower but easier to use since
it doesn't use the debugger.
- Perrin
Re: apache children waits for each other?
Posted by Badai Aqrandista <ba...@hotmail.com>.
>It does look odd. Maybe you are still loading some code before calling
>Apache::DB->init(). Can you post your httpd.conf, or at least the mod_perl
>part of it?
.... start of mod_perl configuration ...
<Perl>
use lib '/i4u/web/elres-mp';
# require Apache::DB;
# Apache::DB->init;
</Perl>
PerlRequire /i4u/web/elres-mp/etc/startup.pl
PerlModule Apache::DProf
PerlModule ELRes::ELRes
PerlModule ELRes::ApacheHandler
PerlTransHandler ELRes::ApacheHandler->trans_handler
PerlLogHandler ELRes::ApacheHandler->log_handler
PerlCleanupHandler ELRes::ApacheHandler->cleanup_handler
... end mod_perl configuration ...
the startup.pl script loads and require's all modules under ELRes::
namespace on startup time, as suggested in mod_perl documentation...
>eval { $app->run }; if ($@) { $r->log_error("Server error: $@");
>$app->send_error(err => $@); }
>That's why I think you are loading some code before initializing the
>debugger -- when it doesn't know about those other subs, they don't show up
>in the profile.
the object that $app refer to is dynamically decided based on the hostname,
for example:
a.example.com ---> $app = ELRes::Property->new()
b.example.com ---> $app = ELRes::Property->new()
c.example.com ---> $app = ELRes::Distributor->new()
both ELRes::Property and ELRes::Distributor are children of
ELRes::TopLevelEntity and have run() method...
does this prevents the debugger to know which subroutine should be profiled?
all the test that I've done is on the hostname whose object is of class
ELRes::Property...
Anyway,
Hmmm... Where can I learn more about the debugger?
I didn't know profiling needs debugger, not until I was told about it in
this mailinglist...
Cheers...
---
Badai Aqrandista
Cheepy (?)
_________________________________________________________________
SEEK: Over 80,000 jobs across all industries at Australia's #1 job site.
http://ninemsn.seek.com.au?hotmail
Re: apache children waits for each other?
Posted by Perrin Harkins <pe...@elem.com>.
Badai Aqrandista wrote:
> As I said previously, the handler itself tops everything... I don't know
> why it happens...
It does look odd. Maybe you are still loading some code before calling
Apache::DB->init(). Can you post your httpd.conf, or at least the
mod_perl part of it?
> If I use Devel::Timer to profile its parts, I found
> that this block takes the most time to run (99.98%):
>
> eval { $app->run };
> if ($@) {
> $r->log_error("Server error: $@");
> $app->send_error(err => $@);
> }
That's pretty much the whole thing, isn't it?
> But the call should not be included in the handler's ExclSec time,
> shouldn't it?
That's why I think you are loading some code before initializing the
debugger -- when it doesn't know about those other subs, they don't show
up in the profile.
- Perrin
Re: apache children waits for each other?
Posted by Badai Aqrandista <ba...@hotmail.com>.
>Okay, send it again with the top 10, sorted by -r then. Maybe we can
>make more suggestions.
Exclusive Times
%Time ExclSec CumulS #Calls sec/call Csec/c Name
20.5 14.30 15.222 1 14.305 15.222 ELRes::ApacheHandler::handler
0.37 0.261 0.261 99576 0.0000 0.0000 Date::Simple::as_iso
0.22 0.151 0.153 20364 0.0000 0.0000
Params::Validate::_check_regex_fro
m_xs
0.17 0.117 0.117 6580 0.0000 0.0000 Storable::mretrieve
0.13 0.093 0.093 27854 0.0000 0.0000 Date::Simple::_add
0.09 0.065 0.065 2900 0.0000 0.0000 ELRes::Entity::DESTROY
0.08 0.058 0.058 34766 0.0000 0.0000 Date::Simple::DESTROY
0.08 0.054 0.054 41678 0.0000 0.0000 Date::Simple::_compare
0.05 0.035 0.035 12984 0.0000 0.0000 Date::Simple::__ANON__
0.03 0.020 0.020 36 0.0006 0.0005 Template::Context::filter
0.03 0.019 0.019 102 0.0002 0.0002 DBI::common::DESTROY
0.01 0.010 0.010 1 0.0100 0.0100 DynaLoader::bootstrap
0.01 0.010 0.010 1 0.0100 0.0100 ELRes::App::Session::DESTROY
0.01 0.010 -0.000 2 0.0050 -
ELRes::ApacheHandler::trans_handle
0.0000 r
0.01 0.010 0.010 25 0.0004 0.0004 DBD::mysql::db::_login
As I said previously, the handler itself tops everything... I don't know why
it happens... If I use Devel::Timer to profile its parts, I found that this
block takes the most time to run (99.98%):
eval { $app->run };
if ($@) {
$r->log_error("Server error: $@");
$app->send_error(err => $@);
}
But the call should not be included in the handler's ExclSec time, shouldn't
it? The eval doesn't make the call included in the ExclSec time because I've
tested without it and it still use 99.98% of the time...
Wierd eh?
Thanks a lot...
---
Badai Aqrandista
Cheepy (?)
_________________________________________________________________
Sell your car for $9 on carpoint.com.au
http://www.carpoint.com.au/sellyourcar
Re: apache children waits for each other?
Posted by Perrin Harkins <pe...@elem.com>.
On Wed, 2005-08-17 at 10:15 +1000, Badai Aqrandista wrote:
> >Was the previous output sorted with the -r flag though?
>
> No.
Okay, send it again with the top 10, sorted by -r then. Maybe we can
make more suggestions.
- Perrin
Re: profiling DBI (was Re: apache children waits for each other?)
Posted by "Philip M. Gollucci" <pg...@p6m7g8.com>.
Perrin Harkins wrote:
> Malcolm J Harwood wrote:
>
>> Somehow I managed to miss the existence of DBI::Profile. Has anyone
>> gotten this working with Apache::DBI (so that it aggregates across the
>> entire lifetime of the DB connection)?
>
>
> Just use DBI::ProfileDumper::Apache.
I don't believe thats been ported mp2.
Also doesn't look that hard to do so.
--
END
------------------------------------------------------------
What doesn't kill us can only make us stronger.
Nothing is impossible.
Philip M. Gollucci (pgollucci@p6m7g8.com) 301.254.5198
Consultant / http://p6m7g8.net/Resume/
Senior Developer / Liquidity Services, Inc.
http://www.liquidityservicesinc.com
http://www.liquidation.com
http://www.uksurplus.com
http://www.govliquidation.com
http://www.gowholesale.com
Re: profiling DBI (was Re: apache children waits for each other?)
Posted by Perrin Harkins <pe...@elem.com>.
Malcolm J Harwood wrote:
> Somehow I managed to miss the existence of DBI::Profile. Has anyone gotten
> this working with Apache::DBI (so that it aggregates across the entire
> lifetime of the DB connection)?
Just use DBI::ProfileDumper::Apache.
- Perrin
profiling DBI (was Re: apache children waits for each other?)
Posted by Malcolm J Harwood <mj...@liminalflux.net>.
> >You can look at the tuning information on http://modperlbook.org/ for
> >some advice as well. And don't forget that there is a profiler for DBI
> >queries that comes with DBI.
Somehow I managed to miss the existence of DBI::Profile. Has anyone gotten
this working with Apache::DBI (so that it aggregates across the entire
lifetime of the DB connection)?
--
"You heard Mr. Garibaldi. You did the right thing."
"Darling, I did the necessary thing. That is not always the same as
the right thing."
- Janice and Laura Rosen in Babylon 5:"The Quality of Mercy"
Re: apache children waits for each other?
Posted by Badai Aqrandista <ba...@hotmail.com>.
>Was the previous output sorted with the -r flag though?
No.
>There's no trick, it's the same as in any language: find out what the
>slow parts are by using a profiler and tinker with them until they are
>faster.
>
>You can look at the tuning information on http://modperlbook.org/ for
>some advice as well. And don't forget that there is a profiler for DBI
>queries that comes with DBI.
I've done the database profiling as well...
Thanks for your help...
---
Badai Aqrandista
Cheepy (?)
_________________________________________________________________
REALESTATE: biggest buy/rent/share listings
http://ninemsn.realestate.com.au
Re: apache children waits for each other?
Posted by Perrin Harkins <pe...@elem.com>.
On Wed, 2005-08-17 at 09:23 +1000, Badai Aqrandista wrote:
> I got the previous output because I put in the debugger initialization...
Was the previous output sorted with the -r flag though? It just seems
so unlikely that Params::Validate would take significant time.
If it was sorted with -r, it looks like your use of memcached is working
against you. Maybe you have a slow network connection to the memcached
server, or are just storing too much in it. I'd suggest you look into
using a local cache instead, like Cache::FastMmap. This will be faster,
although the cache will not be shared between machines.
> Without initialization (Apache::DB->init)
No point in looking at that. It's missing most of your code.
> How should I interpret these outputs and what should I do about it to
> improve my app's speed? This is the first time for me to optimize a large
> perl codebase... So I need a bit of guidance here...
There's no trick, it's the same as in any language: find out what the
slow parts are by using a profiler and tinker with them until they are
faster.
You can look at the tuning information on http://modperlbook.org/ for
some advice as well. And don't forget that there is a profiler for DBI
queries that comes with DBI.
- Perrin
Re: apache children waits for each other?
Posted by Badai Aqrandista <ba...@hotmail.com>.
>I also suspect that you didn't initialize the debugger before compiling
>your code, since none of your code appears in this output.
I got the previous output because I put in the debugger initialization...
Without initialization (Apache::DB->init), this is the top 5 output before
the cache is filled (with dprofpp -r):
Exclusive Times
%Time ExclSec CumulS #Calls sec/call Csec/c Name
33.5 23.74 5.224 1 23.747 5.2236 ELRes::ApacheHandler::handler
1.09 0.772 0.772 7869 0.0001 0.0001 Storable::mretrieve
0.87 0.615 0.615 1418 0.0004 0.0004 Storable::net_mstore
0.70 0.495 0.495 1524 0.0003 0.0003 DBD::mysql::db::_login
0.54 0.380 0.380 2902 0.0001 0.0001 DBI::_new_sth
0.51 0.359 0.388 8852 0.0000 0.0000 DBI::common::DESTROY
and this one is after it's filled:
Exclusive Times
%Time ExclSec CumulS #Calls sec/call Csec/c Name
16.5 13.85 14.780 1 13.857 14.780 ELRes::ApacheHandler::handler
0.21 0.174 0.174 34766 0.0000 0.0000 Date::Simple::DESTROY
0.20 0.170 0.170 6580 0.0000 0.0000 Storable::mretrieve
0.19 0.162 0.162 99576 0.0000 0.0000 Date::Simple::as_iso
0.14 0.115 0.115 27854 0.0000 0.0000 Date::Simple::_add
0.11 0.090 0.090 25 0.0036 0.0036 DBD::mysql::db::_login
How should I interpret these outputs and what should I do about it to
improve my app's speed? This is the first time for me to optimize a large
perl codebase... So I need a bit of guidance here...
Thanks a lot...
---
Badai Aqrandista
Cheepy (?)
_________________________________________________________________
Your opinion counts..for your chance to win a Mini Cooper click here
http://www.qualifiedopinions.com/joinup.php?source=hotmail
Re: apache children waits for each other?
Posted by Perrin Harkins <pe...@elem.com>.
Badai Aqrandista wrote:
> Yes, I'm aware of that. The problem is that the database structure is a
> reminiscent of the old version, which creates one database per client.
> It used to be only 40 clients and one web server. Now we have 300+
> clients (=300+ databases) and 2 web servers. I always get 'Too many
> connections' error when I test the application with more than 100
> concurrent request. Any other ideas?
If you can use Apache::Session::Memcached, why can't you use
Apache::Session::MySQL with a common database login? That will mean one
connection per apache process. You can also raise the connection limit
on your database server if it isn't dying under the load.
> Now, This is the top 5 output of 'dprofpp tmon.out' before data is cached:
You should run dprofpp with the -r flag. This output doesn't even show
the database work in the top 5, and that is normally the top area where
time gets spent. I also suspect that you didn't initialize the debugger
before compiling your code, since none of your code appears in this
output. Params::Validate::_validate is very fast and wouldn't be
showing up this high if the rest of your code was getting profiled.
- Perrin
Re: apache children waits for each other?
Posted by Badai Aqrandista <ba...@hotmail.com>.
>
>Badai Aqrandista wrote:
>>My mod_perl web app uses memcached to cache most of the (MySQL) database
>>query results and as the session storage (Apache::Session::Memcached).
>
>Would it be a problem for your application if you suddenly lost all of your
>session data? That could happen with memcached.
Yes, I'm aware of that. The problem is that the database structure is a
reminiscent of the old version, which creates one database per client. It
used to be only 40 clients and one web server. Now we have 300+ clients
(=300+ databases) and 2 web servers. I always get 'Too many connections'
error when I test the application with more than 100 concurrent request. Any
other ideas?
>As for how to find what is making your application slow, use Apache::DProf.
> That is the best way to find the bottleneck.
I have, and I have optimised it (with caches and indexes).
Now, This is the top 5 output of 'dprofpp tmon.out' before data is cached:
Exclusive Times
%Time ExclSec CumulS #Calls sec/call Csec/c Name
14.4 4.189 9.431 16511 0.0003 0.0006 Cache::Memcached::_load_multi
12.5 3.639 3.639 693 0.0053 0.0053
Compress::Zlib::deflateStream::def
late
8.94 2.589 3.018 46501 0.0001 0.0001 Params::Validate::_validate
7.68 2.225 4.025 17582 0.0001 0.0002
Cache::Memcached::__ANON__[/usr/sh
are/perl5/Cache/Memcached.pm:667]
4.20 1.217 1.217 16511 0.0001 0.0001
Cache::Memcached::__ANON__[/usr/sh
are/perl5/Cache/Memcached.pm:691]
... and this is after the cache is filled up ...
Exclusive Times
%Time ExclSec CumulS #Calls sec/call Csec/c Name
22.5 3.752 6.911 15141 0.0002 0.0005 Cache::Memcached::_load_multi
15.6 2.602 2.939 38569 0.0001 0.0001 Params::Validate::_validate
9.56 1.595 2.242 15173 0.0001 0.0001
Cache::Memcached::__ANON__[/usr/sh
are/perl5/Cache/Memcached.pm:667]
5.50 0.917 0.917 15141 0.0001 0.0001
Cache::Memcached::__ANON__[/usr/sh
are/perl5/Cache/Memcached.pm:691]
4.05 0.676 0.676 102787 0.0000 0.0000 Class::Accessor::get
So, as you see, the subroutines that takes most of the time are realy needed
(to speed up the code and to enforce code correctness). But I still need
more speed (yes, I'm obsessed with speed :p... Hehe... It's just the old
version -a bunch of cgi scripts put together- runs twice as fast as the new
version -a modular OO structure-, and my boss wasn't pleased to hear
that...)
Any ideas?
Anyway, thanks for your replies...
---
Badai Aqrandista
Cheepy (?)
_________________________________________________________________
Dating? Try Lavalife get 7 days FREE! Sign up NOW.
http://www.lavalife.com/clickthru/clickthru.act?id=ninemsn&context=an99&a=20233&locale=en_AU&_t=33473
Re: apache children waits for each other?
Posted by Perrin Harkins <pe...@elem.com>.
Badai Aqrandista wrote:
> My mod_perl web app uses memcached to cache most of the (MySQL) database
> query results and as the session storage (Apache::Session::Memcached).
Would it be a problem for your application if you suddenly lost all of
your session data? That could happen with memcached.
As for how to find what is making your application slow, use
Apache::DProf. That is the best way to find the bottleneck.
- Perrin
RE: apache children waits for each other? (was: Re: web application speed
problem)
Posted by Badai Aqrandista <ba...@hotmail.com>.
UPDATE: I hadn't done much testing when I wrote this... Well, after a bit
more testing they don't wait for each other apparently...
Sorry for filling up your mailbox...
But the actual question remains: How to optimize this web app? I'll send
more questions later with more details...
Thanks for your time...
>Hi all,
>
>My mod_perl web app uses memcached to cache most of the (MySQL) database
>query results and as the session storage (Apache::Session::Memcached). When
>doing performance tests with httperf, I found that apache processes waits
>for each other. I mean: all requests are accepted (apache forks off lots of
>children), then processed (no process finishes), and after a while, all the
>processes finish almost at the same time...
>
....
---
Badai Aqrandista
Cheepy (?)
_________________________________________________________________
SEEK: Over 80,000 jobs across all industries at Australia's #1 job site.
http://ninemsn.seek.com.au?hotmail
apache children waits for each other? (was: Re: web application speed problem)
Posted by Badai Aqrandista <ba...@hotmail.com>.
Hi all,
My mod_perl web app uses memcached to cache most of the (MySQL) database
query results and as the session storage (Apache::Session::Memcached). When
doing performance tests with httperf, I found that apache processes waits
for each other. I mean: all requests are accepted (apache forks off lots of
children), then processed (no process finishes), and after a while, all the
processes finish almost at the same time...
If I put warns in the apache handler's start and end, it gives this output
(edited for brevity, md5 hash in the end is the session ids, just disregard
them):
... output from 90+ of other warns...
Start process 30324: Tue, 16 Aug 2005 11:00:23
EST,af263d03f531584c4d05c7f3df55c7d4
Start process 30326: Tue, 16 Aug 2005 11:00:23
EST,931f8a3a2de409fb3285c3bd038ec7b6
Start process 30325: Tue, 16 Aug 2005 11:00:23
EST,48ca7271dab9037f3a7426a6a68789e5
Start process 30314: Tue, 16 Aug 2005 11:00:23
EST,259709a683879491498cb81055466948
... after a while, suddenly the following output comes out ...
End process 29803: Tue, 16 Aug 2005 11:05:47
EST,dc7e7c893f0ee15b0b6c94d061f9493c
End process 30337: Tue, 16 Aug 2005 11:05:49
EST,73f520865a13715e02d183c5cbb920d4
End process 30002: Tue, 16 Aug 2005 11:05:49
EST,cbfb00260afade004d3cc3086f425272
End process 30003: Tue, 16 Aug 2005 11:05:51
EST,b84da97b68df5f54e336a7767d64a1f8
... output of other 90+ warns ...
It's like they are not processed concurrently. I wonder why does it happen?
Is it memcached? Or is it how apache works? Or is it because of the
benchmarking application that I use?
Thanks...
---
Badai Aqrandista
Cheepy (?)
_________________________________________________________________
SEEK: Over 80,000 jobs across all industries at Australia's #1 job site.
http://ninemsn.seek.com.au?hotmail
Re: web application speed problem with OO modules
Posted by Badai Aqrandista <ba...@hotmail.com>.
>
>First, give dprofpp the flag that makes it sort by wall time, not by CPU
>time. Otherwise, things that take most of the time in the average program,
>like I/O, will appear to be very fast.
>
>Second, make sure you initialize the debugger before loading any of the
>code that you want to profile, like this:
><Perl>
> require Apache::DB;
> Apache::DB->init;
></Perl>
>
>Any code that you compile (e.g. use from startup.pl) before this will not
>get profiled at all.
>
>- Perrin
Thanks for the replies...
I'll give it a try...
---
Badai Aqrandista
Cheepy (?)
_________________________________________________________________
SEEK: Over 80,000 jobs across all industries at Australia's #1 job site.
http://ninemsn.seek.com.au?hotmail
Re: web application speed problem with OO modules
Posted by Perrin Harkins <pe...@elem.com>.
Badai Aqrandista wrote:
> I doubt the ApacheHandler::handler subroutine runs that long, because it
> only shows the accumulated time used by other subroutines it calls. That
> makes the profile isn't really accurate. How do I make it to show the
> statistics of the subroutines called from the handler?
First, give dprofpp the flag that makes it sort by wall time, not by CPU
time. Otherwise, things that take most of the time in the average
program, like I/O, will appear to be very fast.
Second, make sure you initialize the debugger before loading any of the
code that you want to profile, like this:
<Perl>
require Apache::DB;
Apache::DB->init;
</Perl>
Any code that you compile (e.g. use from startup.pl) before this will
not get profiled at all.
- Perrin
Re: web application speed problem with OO modules
Posted by Badai Aqrandista <ba...@hotmail.com>.
Thanks for replying...
I have done some profiling with Apache::DProfile, and this is the output of
'dprofpp tmon.out':
Total Elapsed Time = 160.2141 Seconds
User+System Time = 13.07412 Seconds
Exclusive Times
%Time ExclSec CumulS #Calls sec/call Csec/c Name
92.7 12.12 13.575 1 12.127 13.574 ELRes::ApacheHandler::handler
3.13 0.409 0.409 90360 0.0000 0.0000 Date::Simple::as_iso
2.20 0.288 0.366 20364 0.0000 0.0000
Params::Validate::_check_regex_fro
m_xs
1.53 0.200 0.200 29966 0.0000 0.0000 Date::Simple::DESTROY
1.35 0.177 0.177 6565 0.0000 0.0000 Storable::mretrieve
0.66 0.086 0.086 36878 0.0000 0.0000 Date::Simple::_compare
0.58 0.076 0.076 1761 0.0000 0.0000 DBD::_::db::prepare_cached
0.46 0.060 0.060 24782 0.0000 0.0000 Date::Simple::_add
0.32 0.042 0.042 2900 0.0000 0.0000 ELRes::Entity::DESTROY
0.25 0.033 0.033 3700 0.0000 0.0000 DBI::common::FETCH
0.18 0.023 0.023 3692 0.0000 0.0000 DBI::st::fetch
0.12 0.016 0.016 1756 0.0000 0.0000 ELRes::DB::DESTROY
0.08 0.011 0.011 9528 0.0000 0.0000 Date::Simple::__ANON__
0.08 0.010 0.010 1 0.0100 0.0100 DynaLoader::bootstrap
0.08 0.010 0.010 2 0.0050 0.0050 DBD::mysql::db::_login
I doubt the ApacheHandler::handler subroutine runs that long, because it
only shows the accumulated time used by other subroutines it calls. That
makes the profile isn't really accurate. How do I make it to show the
statistics of the subroutines called from the handler?
Thank you...
---
Badai Aqrandista
Cheepy (?)
>From: "Philip M. Gollucci" <pg...@p6m7g8.com>
>To: Badai Aqrandista <ba...@hotmail.com>
>CC: modperl@perl.apache.org
>Subject: Re: web application speed problem with OO modules
>Date: Mon, 15 Aug 2005 00:32:09 -0400
>
>>(I only try to improve the speed of the code, so the use of proxy
>>shouldn't be relevant here - I think, I could be wrong).
>You might try using
>
>Apache::DProf
>and/or
>Apache::SmallProf
>
>available here to see where your code is spending its time
>
>http://search.cpan.org/~fwiles/Apache-DB
_________________________________________________________________
Low rate ANZ MasterCard. Apply now!
http://clk.atdmt.com/MAU/go/msnnkanz0030000006mau/direct/01/ Must be over
18 years.
Re: web application speed problem with OO modules
Posted by "Philip M. Gollucci" <pg...@p6m7g8.com>.
> (I only try to improve the speed of the code, so the use of proxy
> shouldn't be relevant here - I think, I could be wrong).
You might try using
Apache::DProf
and/or
Apache::SmallProf
available here to see where your code is spending its time
http://search.cpan.org/~fwiles/Apache-DB