You are viewing a plain text version of this content. The canonical link for it is here.
Posted to modperl@perl.apache.org by so...@sofcorp.com on 2005/01/14 21:43:44 UTC
[BENCHMARK] Server Resources
First let me apologize if this is not the appropriate forum for my question.
I thought it would be because the issue has to do with
apache/modperl/mysql performance.
We have recently added a site to our server which hosts 30 other sites.
The new site uses modperl & MySQL. After adding the new site, the server
we were on (shared server at a national server farm) had major resource
problems (too many connections & load avg up to 30 not infrequently) &
eventually crashed. We moved to a new dedicated server which gave us more
resources, but we're still experiencing intermittent sluggishness - load
avg approaching 10 - and this is before this new site really becomes
active. Maybe 20 logins per day currently - in 10 days we'll have ~ 4000.
I've made changes to Apache config, following suggestions in the mod_perl
performance tuning docs:
MinSpareServers 5
MaxSpareServers 10
MaxClients=75
MaxRequestsPerChild 500
I've made efforts to optimize the database & queries, and have made
changes to the my.cnf file following suggestions in the High Performance
MySQL O'Reilly book & a couple of postings that seemed to have some
similarity to our situation:
set-variable = key_buffer_size=128M
set-variable = table_cache=1024
set-variable = join_buffer=1M
set-variable = sort_buffer=2M
set-variable = record_buffer=1M
set-variable = wait_timeout=20
set-variable = thread_cache=8
But before I go any further with testing & modifying configurations &
perhaps code, I'd like to have some good idea that the server we currently
are running on should be able to handle this new site & the 30 others - 2
running WebGUI, 12 use MySQL (4-5 fairly heavily), 4 modperl.
The server is a Pentium 3 800 MHz with 768M RAM (of which we can use ~640,
according to the server sysadmin)
It's running Ensim 4.0 with Fedora Core 1 with Embedded Perl version
v5.8.1 for Apache/1.3.31 (Unix) (Red-Hat/Linux) mod_perl/1.29 PHP/4.3.8
FrontPage/5.0.2.2635 mod_ssl/2.8.18 OpenSSL/0.9.7a
MySQL version 3.23.58
Any constructive suggestions are very welcome.
Thanks.
Sys
RE: [BENCHMARK] Server Resources
Posted by Todd Finney <tf...@boygenius.com>.
At 02:02 AM 1/16/2005 +0000, Tom Gazzini wrote:
>How does this help if the server hosts only one sites which is base on, say,
>Mason, and where every page request would require mod_perl?
This is covered in delightfully interminable detail in the "Choosing the
Right Strategy" section of the guide, but I'll summarize.
When you're only running a mod_perl server, you wind up with a 30MB process
sitting in memory, waiting around for some putz in Iowa on a 28.8 dialup
line to download the output, including the script output, as well as the
images and the useless Flash navigation menu, etc. While it's doing that,
it can't do anything else.
When you put a 3MB proxy process in front of it, two things happen:
- The mod_perl process is responsible for *only* the dynamic
content. Everything else is served directly from the lightweight
server. That allows your server to do more work with the resources that it
has.
- The mod_perl process no longer has to wait for the download to
complete. The proxy server takes the output from the mod_perl server
instantly, and sends it out over the wire to the client. This also allows
the server to do more work with the resources that it has.
Those are the two big benefits, there are others that are covered in the guide.
<http://perl.apache.org/docs/1.0/guide/strategy.html#Adding_a_Proxy_Server_in_http_Accelerator_Mode>
I've been running this setup for a few years now, it's been good to
me. Highly recommended.
Re: [BENCHMARK] Server Resources
Posted by Perrin Harkins <pe...@elem.com>.
On Sat, 2005-01-15 at 18:31 -0700, sofadmin@sofcorp.com wrote:
> Would you recommend squid or mod_proxy?
I would recommend mod_proxy because you already know how to run apache,
and it can run useful things like mod_rewrite, mod_ssl, mod_auth_tkt,
etc. I benchmarked them a VERY long time ago and found little
difference, but I'd love to see someone do an updated benchmark.
- Perrin
RE: [BENCHMARK] Server Resources
Posted by Tom Gazzini <to...@barzakh.co.uk>.
How does this help if the server hosts only one sites which is base on, say,
Mason, and where every page request would require mod_perl?
-Tom
> -----Original Message-----
> From: sofadmin@sofcorp.com [mailto:sofadmin@sofcorp.com]
> Sent: 16 January 2005 01:31
> To: modperl@perl.apache.org
> Subject: Re: [BENCHMARK] Server Resources
>
> .
> >
> > Probably the biggest bang for the buck with the least effort or code
> > changes is to set up a lightweight (i.e. non-mod_perl) reverse proxy
> httpd
> > in front of the httpd that is running mod_perl:
> >
>
> Would you recommend squid or mod_proxy?
>
Re: [BENCHMARK] Server Resources
Posted by so...@sofcorp.com.
.
>
> Probably the biggest bang for the buck with the least effort or code
> changes is to set up a lightweight (i.e. non-mod_perl) reverse proxy httpd
> in front of the httpd that is running mod_perl:
>
Would you recommend squid or mod_proxy?
Re: [BENCHMARK] Server Resources
Posted by Stas Bekman <st...@stason.org>.
Larry Leszczynski wrote:
[...]
> Probably the biggest bang for the buck with the least effort or code
> changes is to set up a lightweight (i.e. non-mod_perl) reverse proxy httpd
> in front of the httpd that is running mod_perl:
>
> http://perl.apache.org/docs/1.0/guide/strategy.html#Adding_a_Proxy_Server_in_http_Accelerator_Mode
>
>
> Lots of other good tuning info at:
>
> http://perl.apache.org/docs/1.0/guide/performance.html
And much more detailed and polished at:
http://modperlbook.org/html/part2.html
In fact since the book is now freely available, someone could backport the
text in its completeness into the perl.apache.org guide. At the beginning
I was backporting all the new stuff I wrote for the book, but then it
became too time consuming, so I've stopped doing that.
--
__________________________________________________________________
Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide ---> http://perl.apache.org
mailto:stas@stason.org http://use.perl.org http://apacheweek.com
http://modperlbook.org http://apache.org http://ticketmaster.com
Re: [BENCHMARK] Server Resources
Posted by Larry Leszczynski <la...@emailplus.org>.
On Fri, 14 Jan 2005 sofadmin@sofcorp.com wrote:
> We have recently added a site to our server which hosts 30 other sites.
> The new site uses modperl & MySQL. After adding the new site, the server
> we were on (shared server at a national server farm) had major resource
> problems (too many connections & load avg up to 30 not infrequently) &
> eventually crashed. We moved to a new dedicated server which gave us more
> resources, but we're still experiencing intermittent sluggishness - load
> avg approaching 10 - and this is before this new site really becomes
> active. Maybe 20 logins per day currently - in 10 days we'll have ~ 4000.
Probably the biggest bang for the buck with the least effort or code
changes is to set up a lightweight (i.e. non-mod_perl) reverse proxy httpd
in front of the httpd that is running mod_perl:
http://perl.apache.org/docs/1.0/guide/strategy.html#Adding_a_Proxy_Server_in_http_Accelerator_Mode
Lots of other good tuning info at:
http://perl.apache.org/docs/1.0/guide/performance.html
HTH,
Larry Leszczynski
Re: [BENCHMARK] Server Resources
Posted by Rob Bloodgood <ro...@exitexchange.com>.
Skylos wrote:
>My first suggestion based on a migration the company I work for did
>not too long ago would be to use apache2 in thread mode. There was a
>staggering drop in system memory resources consumed when we made the
>shift - from most of a gigabyte to only a few hundred megs! The
>site's response speed picked up too.
>
>Also, to compare, I work with an apache 1.3 site that has alot of cgi
>perl script on it. With the idea that shifting to mod_perl registry
>mode would cause this site to go faster, I modified the configuration.
> And watched the system load average rapidly climb into the
>multiple-hundreds! Its a tilt of the hat to BSD that it didn't crash
>right there. But I was able to kill apache (-9 -9 -9!!) and restore
>the cgi scripts to normal mod_cgi handling. What happened? Memory
>usage. Each copy of the apache had those big cgi scripts loaded in.
>And this very busy server had ALOT of processes running
>simultaneously.
>
>
If you don't need KeepAlives, (like for mostly dynamic output and few
images or a seperate image server), you should check out lingerd:
http://www.iagora.com/about/software/lingerd/.
On my site, which was running at the time with something like 85
concurrent apache processes to keep up with the requests, I turned on
lingerd and the number of processes running dropped to 10-12. (saved us
upgrading the box for another year :). The memory usage and load
average dropped accordingly, but naturally YMMV.
L8r,
Rob
Re: [BENCHMARK] Server Resources
Posted by Perrin Harkins <pe...@elem.com>.
On Fri, 2005-01-14 at 13:08 -0800, Skylos wrote:
> My first suggestion based on a migration the company I work for did
> not too long ago would be to use apache2 in thread mode. There was a
> staggering drop in system memory resources consumed when we made the
> shift - from most of a gigabyte to only a few hundred megs! The
> site's response speed picked up too.
No offense, but I think there may be some other reason for that. All
reports so far say that running in threaded mode uses significantly more
memory than pre-fork because it defeats the copy-on-write sharing.
> Also, to compare, I work with an apache 1.3 site that has alot of cgi
> perl script on it. With the idea that shifting to mod_perl registry
> mode would cause this site to go faster, I modified the configuration.
> And watched the system load average rapidly climb into the
> multiple-hundreds!
That's probably because you didn't tune the server settings for the
increased memory needed by mod_perl. When you run mod_perl, you have to
adjust your MaxClients setting and use something like Apache::SizeLimit
to limit the size of processes. This keeps you from going over the
available RAM. A good setup for a site that has some legacy CGI to run
as well is to make the CGI server run mod_proxy and proxy the mod_perl
requests back to a separate server that just runs mod_perl. Running a
small number of mod_perl processes will still give you better
performance than a large number of CGI processes because the speed gains
are so large.
- Perrin
Re: [BENCHMARK] Server Resources
Posted by Skylos <sk...@gmail.com>.
Way not enough information. Like, you can host 30 websites on a
386sx-25 with 64 megs of ram - if they're low
demand enough. It really has no relevancy to the number of sites.
Its all about how many file presentations you're handling, and more
importantly, how many dynamic script presentations you're handling.
Skylos
On Fri, 14 Jan 2005 14:42:05 -0700 (MST), sofadmin@sofcorp.com
<so...@sofcorp.com> wrote:
> Thanks for the suggestions. We are planning on migrating to Apache 2.0,
> but I need to read up on the implications to the various sites first.
>
> Do you think this server has enough resources to handle our sites without
> trouble?
>
> Sys
>
> >>
> >> But before I go any further with testing & modifying configurations &
> >> perhaps code, I'd like to have some good idea that the server we
> >> currently
> >> are running on should be able to handle this new site & the 30 others -
> >> 2
> >> running WebGUI, 12 use MySQL (4-5 fairly heavily), 4 modperl.
> >>
> >> The server is a Pentium 3 800 MHz with 768M RAM (of which we can use
> >> ~640,
> >> according to the server sysadmin)
> >>
> >> It's running Ensim 4.0 with Fedora Core 1 with Embedded Perl version
> >> v5.8.1 for Apache/1.3.31 (Unix) (Red-Hat/Linux) mod_perl/1.29 PHP/4.3.8
> >> FrontPage/5.0.2.2635 mod_ssl/2.8.18 OpenSSL/0.9.7a
> >> MySQL version 3.23.58
> >>
> >> Any constructive suggestions are very welcome.
> >>
> >> Thanks.
> >>
> >> Sys
> >>
> >>
> >
>
>
Re: [BENCHMARK] Server Resources
Posted by so...@sofcorp.com.
Thanks for the suggestions. We are planning on migrating to Apache 2.0,
but I need to read up on the implications to the various sites first.
Do you think this server has enough resources to handle our sites without
trouble?
Sys
>>
>> But before I go any further with testing & modifying configurations &
>> perhaps code, I'd like to have some good idea that the server we
>> currently
>> are running on should be able to handle this new site & the 30 others -
>> 2
>> running WebGUI, 12 use MySQL (4-5 fairly heavily), 4 modperl.
>>
>> The server is a Pentium 3 800 MHz with 768M RAM (of which we can use
>> ~640,
>> according to the server sysadmin)
>>
>> It's running Ensim 4.0 with Fedora Core 1 with Embedded Perl version
>> v5.8.1 for Apache/1.3.31 (Unix) (Red-Hat/Linux) mod_perl/1.29 PHP/4.3.8
>> FrontPage/5.0.2.2635 mod_ssl/2.8.18 OpenSSL/0.9.7a
>> MySQL version 3.23.58
>>
>> Any constructive suggestions are very welcome.
>>
>> Thanks.
>>
>> Sys
>>
>>
>
Re: [BENCHMARK] Server Resources
Posted by Skylos <sk...@gmail.com>.
No, i didn't preload the scripts - I'm not familiar with the
technique, and it seems like there's enough dynamic fvariable-writing
and storage of html values done in the scripts that large chunks would
be unfairly 'written' and cause duplication. Its a bad system design
for this volume, it should be filesystem cached static files that
update on change. We're working on that. one project at a time. :)
Skylos
On Fri, 14 Jan 2005 16:16:30 -0500, Michael Peters
<mp...@plusthree.com> wrote:
>
> Skylos wrote:
>
> > Also, to compare, I work with an apache 1.3 site that has alot of cgi
> > perl script on it. With the idea that shifting to mod_perl registry
> > mode would cause this site to go faster, I modified the configuration.
> > And watched the system load average rapidly climb into the
> > multiple-hundreds! Its a tilt of the hat to BSD that it didn't crash
> > right there. But I was able to kill apache (-9 -9 -9!!) and restore
> > the cgi scripts to normal mod_cgi handling. What happened? Memory
> > usage. Each copy of the apache had those big cgi scripts loaded in.
> > And this very busy server had ALOT of processes running
> > simultaneously.
>
> Did you preload the those scripts at server start-up? This would change
> it so that instead of each apache child having it's own copy of each
> script they would have been placed in shared (copy-on-write) memory.
>
> --
> Michael Peters
> Developer
> Plus Three, LP
>
>
Re: [BENCHMARK] Server Resources
Posted by Michael Peters <mp...@plusthree.com>.
Skylos wrote:
> Also, to compare, I work with an apache 1.3 site that has alot of cgi
> perl script on it. With the idea that shifting to mod_perl registry
> mode would cause this site to go faster, I modified the configuration.
> And watched the system load average rapidly climb into the
> multiple-hundreds! Its a tilt of the hat to BSD that it didn't crash
> right there. But I was able to kill apache (-9 -9 -9!!) and restore
> the cgi scripts to normal mod_cgi handling. What happened? Memory
> usage. Each copy of the apache had those big cgi scripts loaded in.
> And this very busy server had ALOT of processes running
> simultaneously.
Did you preload the those scripts at server start-up? This would change
it so that instead of each apache child having it's own copy of each
script they would have been placed in shared (copy-on-write) memory.
--
Michael Peters
Developer
Plus Three, LP
Re: [BENCHMARK] Server Resources
Posted by Skylos <sk...@gmail.com>.
My first suggestion based on a migration the company I work for did
not too long ago would be to use apache2 in thread mode. There was a
staggering drop in system memory resources consumed when we made the
shift - from most of a gigabyte to only a few hundred megs! The
site's response speed picked up too.
Also, to compare, I work with an apache 1.3 site that has alot of cgi
perl script on it. With the idea that shifting to mod_perl registry
mode would cause this site to go faster, I modified the configuration.
And watched the system load average rapidly climb into the
multiple-hundreds! Its a tilt of the hat to BSD that it didn't crash
right there. But I was able to kill apache (-9 -9 -9!!) and restore
the cgi scripts to normal mod_cgi handling. What happened? Memory
usage. Each copy of the apache had those big cgi scripts loaded in.
And this very busy server had ALOT of processes running
simultaneously.
To summarize - apache2 with threads may solve the problem - shifting
to cgi mode for the perl scripts may solve the problem.
Limiting the number of simultaneous apaches will help - but will slow
down the site.
But thats about the extent of what I can say, I defer to any other
more expert perl ghods on the list who may have more intelligent
comments. :)
Skylos
On Fri, 14 Jan 2005 13:43:44 -0700 (MST), sofadmin@sofcorp.com
<so...@sofcorp.com> wrote:
> First let me apologize if this is not the appropriate forum for my question.
> I thought it would be because the issue has to do with
> apache/modperl/mysql performance.
>
> We have recently added a site to our server which hosts 30 other sites.
> The new site uses modperl & MySQL. After adding the new site, the server
> we were on (shared server at a national server farm) had major resource
> problems (too many connections & load avg up to 30 not infrequently) &
> eventually crashed. We moved to a new dedicated server which gave us more
> resources, but we're still experiencing intermittent sluggishness - load
> avg approaching 10 - and this is before this new site really becomes
> active. Maybe 20 logins per day currently - in 10 days we'll have ~ 4000.
>
> I've made changes to Apache config, following suggestions in the mod_perl
> performance tuning docs:
> MinSpareServers 5
> MaxSpareServers 10
> MaxClients=75
> MaxRequestsPerChild 500
>
> I've made efforts to optimize the database & queries, and have made
> changes to the my.cnf file following suggestions in the High Performance
> MySQL O'Reilly book & a couple of postings that seemed to have some
> similarity to our situation:
> set-variable = key_buffer_size=128M
> set-variable = table_cache=1024
> set-variable = join_buffer=1M
> set-variable = sort_buffer=2M
> set-variable = record_buffer=1M
> set-variable = wait_timeout=20
> set-variable = thread_cache=8
>
> But before I go any further with testing & modifying configurations &
> perhaps code, I'd like to have some good idea that the server we currently
> are running on should be able to handle this new site & the 30 others - 2
> running WebGUI, 12 use MySQL (4-5 fairly heavily), 4 modperl.
>
> The server is a Pentium 3 800 MHz with 768M RAM (of which we can use ~640,
> according to the server sysadmin)
>
> It's running Ensim 4.0 with Fedora Core 1 with Embedded Perl version
> v5.8.1 for Apache/1.3.31 (Unix) (Red-Hat/Linux) mod_perl/1.29 PHP/4.3.8
> FrontPage/5.0.2.2635 mod_ssl/2.8.18 OpenSSL/0.9.7a
> MySQL version 3.23.58
>
> Any constructive suggestions are very welcome.
>
> Thanks.
>
> Sys
>
>
Re: [BENCHMARK] Server Resources
Posted by so...@sofcorp.com.
> Hey there
>
> It strikes me that your problem is directly related to memory usage.
> Linux machines have a bad problem in that if your apache children get
> swapped out ->for any reason at all<-, they will lose any shared memory
> that they may have had before being swapped out. When they are swapped
> back in, they take up more RAM than before, so the system swaps out
> another Apache process, and so on, until the system enters the
> "downward spiral of death".
Thanks for the heads up on this -- I've been seeing this happen with the
process size & didn't know what to attribute it to. I implemented
Apache::SizeLimit a couple months ago when we added a WebGUI site -
processes were growing extravagantly. Still haven't found the ideal
setting, but getting closer, I think.
Sys
Re: [BENCHMARK] Server Resources
Posted by Kyle Dawkins <ky...@centralparksoftware.com>.
Hey there
It strikes me that your problem is directly related to memory usage.
Linux machines have a bad problem in that if your apache children get
swapped out ->for any reason at all<-, they will lose any shared memory
that they may have had before being swapped out. When they are swapped
back in, they take up more RAM than before, so the system swaps out
another Apache process, and so on, until the system enters the
"downward spiral of death".
You need to memory-limit your apache processes using Apache::SizeLimit
to make sure that they NEVER go over a certain size. Then you have to
make sure that
MaxClients * Apache::SizeLimit < Available core memory
which will make you miserable because you have MySQL running on the
same machine. I would recommend moving MySQL off if you can, which
will greatly simplify your memory calculations. If you only have 768MB
on your machine, then you need to subtract the OS's basic footprint,
and anything that's running, subtract a bit more for breathing room,
then divide what's left by the Apache::SizeLimit number that you have
found (through experimentation and observation) to be the stable size
of your apache children, and that will give you the MaxClient number.
Then, you pre-load all of your perl modules and as many of your perl
scripts as possible, and you should be fine. I have seen this happen
in almost every mod_perl (<1.99, admittedly) environment I have ever
worked on. It takes some tweaking but once you find the magic numbers,
it will not give you any more problems.
Also, I don't know if this is an option for you, but switching to
FreeBSD made our lives much much easier, from almost any way you look
at it. Food for thought.
I also highly recommend the idea of having two separate servers, a
lightweight tiny apache that handles static content, and proxies
dynamic requests to a heavier, mod_perl apache. You can save huge
amounts of memory that way, and it tends to increase performance
significantly. It's also incredibly easy to set up.
Hope this helps!
Kyle
Central Park Software
> We have recently added a site to our server which hosts 30 other sites.
> The new site uses modperl & MySQL. After adding the new site, the
> server
> we were on (shared server at a national server farm) had major resource
> problems (too many connections & load avg up to 30 not infrequently) &
> eventually crashed. We moved to a new dedicated server which gave us
> more
> resources, but we're still experiencing intermittent sluggishness -
> load
> avg approaching 10 - and this is before this new site really becomes
> active. Maybe 20 logins per day currently - in 10 days we'll have ~
> 4000.
>
> I've made changes to Apache config, following suggestions in the
> mod_perl
> performance tuning docs:
> MinSpareServers 5
> MaxSpareServers 10
> MaxClients=75
> MaxRequestsPerChild 500
>
> I've made efforts to optimize the database & queries, and have made
> changes to the my.cnf file following suggestions in the High
> Performance
> MySQL O'Reilly book & a couple of postings that seemed to have some
> similarity to our situation:
> set-variable = key_buffer_size=128M
> set-variable = table_cache=1024
> set-variable = join_buffer=1M
> set-variable = sort_buffer=2M
> set-variable = record_buffer=1M
> set-variable = wait_timeout=20
> set-variable = thread_cache=8
>
> But before I go any further with testing & modifying configurations &
> perhaps code, I'd like to have some good idea that the server we
> currently
> are running on should be able to handle this new site & the 30 others
> - 2
> running WebGUI, 12 use MySQL (4-5 fairly heavily), 4 modperl.
>
> The server is a Pentium 3 800 MHz with 768M RAM (of which we can use
> ~640,
> according to the server sysadmin)
>
> It's running Ensim 4.0 with Fedora Core 1 with Embedded Perl version
> v5.8.1 for Apache/1.3.31 (Unix) (Red-Hat/Linux) mod_perl/1.29 PHP/4.3.8
> FrontPage/5.0.2.2635 mod_ssl/2.8.18 OpenSSL/0.9.7a
> MySQL version 3.23.58
>
> Any constructive suggestions are very welcome.
>
> Thanks.
>
> Sys
>