You are viewing a plain text version of this content. The canonical link for it is here.
Posted to modperl@perl.apache.org by Badai Aqrandista <ba...@hotmail.com> on 2005/08/23 02:20:45 UTC
swamped with connection?
Hi all, it's me again :D
I am still trying to improve my company's webapp performance. I'm testing it
with httperf and autobench. The application seems to be able to respond when
hammered by 20 connections per second and 10 calls per connection. But then,
it doesn't respond to any request when the connection rate is raised to 40
(with 10 calls per connection) and above.
What does the apache treats as a request (hence forks another child)?
- every new connection, or
- every http request regardless of it comes with a new connection or comes
through an existing connection
Does anyone have any idea what's going on?
My only guess is because the connections goes over the MaxClients count
while the existing apache children are still processing the previous
requests. But why, then, none of the requests after 40 conn/sec is
responded?
I'll give more detail if requested.
THANKS A LOT!!!
---
Badai Aqrandista
Cheepy (?)
_________________________________________________________________
REALESTATE: biggest buy/rent/share listings
http://ninemsn.realestate.com.au
Re: swamped with connection?
Posted by Torsten Foertsch <to...@gmx.net>.
On Tuesday 23 August 2005 21:08, Philip M. Gollucci wrote:
> One comment might be to add an option to turn it off even if the newer
> Smaps support is present?
I have also thought of it. Sounds sound.
I'll send a patch soon.
Torsten
Re: swamped with connection?
Posted by "Philip M. Gollucci" <pg...@p6m7g8.com>.
Torsten Foertsch wrote:
> On Tuesday 23 August 2005 14:23, Perrin Harkins wrote:
>
>>However, you should be aware that a few months back we discovered that
>>our methods for measuring shared memory didn't work very well on Linux
>>2.4 kernels and don't really work at all on 2.6 kernels, so there may be
>>more sharing (via copy-on-write) going on than you can see here.
>
>
> See also
> http://marc.theaimsgroup.com/?l=apache-modperl&m=112343986910467&w=2
Speaking of which, is there any reason not to commit this?
I can finally verify that it works(doesn't break anything) locally on a new box I've got.
One comment might be to add an option to turn it off even if the newer Smaps support is
present?
--
END
------------------------------------------------------------
What doesn't kill us can only make us stronger.
Nothing is impossible.
Philip M. Gollucci (pgollucci@p6m7g8.com) 301.254.5198
Consultant / http://p6m7g8.net/Resume/
Senior Developer / Liquidity Services, Inc.
http://www.liquidityservicesinc.com
http://www.liquidation.com
http://www.uksurplus.com
http://www.govliquidation.com
http://www.gowholesale.com
Re: swamped with connection?
Posted by Torsten Foertsch <to...@gmx.net>.
On Tuesday 23 August 2005 14:23, Perrin Harkins wrote:
> However, you should be aware that a few months back we discovered that
> our methods for measuring shared memory didn't work very well on Linux
> 2.4 kernels and don't really work at all on 2.6 kernels, so there may be
> more sharing (via copy-on-write) going on than you can see here.
See also
http://marc.theaimsgroup.com/?l=apache-modperl&m=112343986910467&w=2
Torsten
Re: swamped with connection?
Posted by Perrin Harkins <pe...@elem.com>.
On Tue, 2005-08-23 at 15:52 +1000, Badai Aqrandista wrote:
> RAM = 700 Mb
> Per process total size = 40 Mb
> Shared memory = 7 Mb
>
> So, the number of processes = (700 - 7) / 33 = 21 processes
>
> So, does that mean it can only accept up to 21 connections?
Yes. If you are running a reverse proxy in front of it, that should be
enough to handle a lot of traffic.
However, you should be aware that a few months back we discovered that
our methods for measuring shared memory didn't work very well on Linux
2.4 kernels and don't really work at all on 2.6 kernels, so there may be
more sharing (via copy-on-write) going on than you can see here.
Looking at the total free memory on your machine when 21 processes are
running may be more useful than just doing the calculation.
- Perrin
Re: maintaining shared memory size (was: Re: swamped
withconnection?)
Posted by Perrin Harkins <pe...@elem.com>.
On Wed, 2005-08-24 at 07:02 -0400, Sean Davis wrote:
> As an aside, are there rules of thumb about what cache works best in various
> situations?
Cache::FastMmap and BerkeleyDB are the fastest by far, but they are
local to one machine. Cache::Memcached or a simple key/value table in a
MySQL server are second fastest, and can be shared between machines.
After that, everything else is a lot slower. Some stats are available
here:
http://cpan.robm.fastmail.fm/cache_perf.html
> Are RDBMS's, if accessible, generally a good solution?
For caching? MySQL is surprisingly fast when handling primary key
requests against a simple table. It's faster than Cache::Cache and
most of the other caching modules.
- Perrin
Re: maintaining shared memory size (was: Re: swamped
withconnection?)
Posted by Sean Davis <sd...@mail.nih.gov>.
On 8/23/05 10:04 PM, "Perrin Harkins" <pe...@elem.com> wrote:
> On Wed, 2005-08-24 at 10:31 +1000, Badai Aqrandista wrote:
>> Anyway, to fix this, I'm trying to make my onw shared memory with
>> Apache::SharedMem.
>
> You can share read-only data by loading it into normal variables during
> startup. You can share read/write data by keeping it in a database
> (including things like BerkeleyDB or Cache::FastMmap) and only reading
> the small parts of it that you need in each process.
As an aside, are there rules of thumb about what cache works best in various
situations? Are RDBMS's, if accessible, generally a good solution? Is
there a document/link giving a comparison?
Thanks,
Sean
Re: maintaining shared memory size (was: Re: swampedwithconnection?)
Posted by Ask Bjørn Hansen <as...@develooper.com>.
On Aug 29, 2005, at 5:11 PM, David Hodgkinson wrote:
> You *do* have KeepAlive off in your httpd, right?
That is one of the great things about perlbal[1]. You can support
KeepAlive without using more resources.
- ask
[1] http://www.danga.com/perlbal/
--
http://www.askbjoernhansen.com/
Re: maintaining shared memory size (was: Re: swampedwithconnection?)
Posted by David Hodgkinson <da...@gmail.com>.
On 30 Aug 2005, at 10:16, Badai Aqrandista wrote:
>>
> I used to use A::S::MySQL, but it created 2 connections for every
> process. This caused 'Too many connections' error. So I tried to
> use memcached. Now that I know how to make one connection per
> process (using database.table identifier on all sql queries), I'll
> probably try to use A::S::MySQL again.
But now you have reduced MaxClients, that shouldn't happen.
Re: maintaining shared memory size (was: Re: swampedwithconnection?)
Posted by Badai Aqrandista <ba...@hotmail.com>.
Hi,
>I see from an earlier post on the mason-users list that your app is
>using HTML::Mason and Apache::Session::Memcached. It seems like you've
>got quite a few variables to juggle in your performance bottleneck
>analysis.
Actually, I am using TT and Mason in this application. TT is used to support
templates from the old version that we did not have time to port to Mason.
The part that needs the speed is the part that uses TT.
>Have you tried switching to, say, Apache::Session::MySQL to see if
>there
> is any difference?
I used to use A::S::MySQL, but it created 2 connections for every process.
This caused 'Too many connections' error. So I tried to use memcached. Now
that I know how to make one connection per process (using database.table
identifier on all sql queries), I'll probably try to use A::S::MySQL again.
>One cheap way to check for memory leaks (most probably introduced by
>your perl code if they exist) is to run httpd -X (non-forking mode) and
>watch the memory usage of httpd while you send some requests.
>
>You may also want to strace the process:
># httpd -X &
># strace -o /tmp/strace.out -p `pidof httpd`
># tail -f /tmp/strace.out
>
>Then hit the server with some requests and watch the strace output.
>This is especially useful for finding I/O or IPC bottlenecks.
Hmmm... Interesting... I'll give it a try...
>good luck
Thanks...
---
Badai Aqrandista
Cheepy (?)
_________________________________________________________________
Dating? Try Lavalife get 7 days FREE! Sign up NOW.
http://www.lavalife.com/clickthru/clickthru.act?id=ninemsn&context=an99&a=20233&locale=en_AU&_t=33473
Re: maintaining shared memory size (was: Re: swampedwithconnection?)
Posted by Tony Clayton <to...@clayton.ca>.
Quoting Badai Aqrandista <ba...@hotmail.com>:
>
> >
> >Then how do you know what to write in C?
> >
>
> I have localized one subroutine that has been heavily called in the
> search
> function. That should be the way to pick the candidate, shouldn't
> it?
>
> I have never done this and I am worried that writing it in C would
> push the
> project's deadline a little bit further. However, does anyone have
> any hint
> on doing this? What are the recommended readings?
>
> Anyhow, I've decrease the MaxClients to 3 and it worked!!! The speed
>
> significantly increases, from average of 110 sec/request to 85
> sec/request.
> But, that's not enough. I need less than 10 sec/request.
>
> Is that possible without changing hardware?
I see from an earlier post on the mason-users list that your app is
using HTML::Mason and Apache::Session::Memcached. It seems like you've
got quite a few variables to juggle in your performance bottleneck
analysis.
Have you tried switching to, say, Apache::Session::MySQL to see if
there
is any difference?
You can also try these measures with your app:
- Use static source mode in mason, and preload components:
http://masonhq.com/docs/manual/Admin.html#static_source_mode
- Preload (some/all) of your perl modules in the httpd parent using the
PerlModule apache directive
- If you are connecting to a database, be sure to use Apache::DBI
One cheap way to check for memory leaks (most probably introduced by
your perl code if they exist) is to run httpd -X (non-forking mode) and
watch the memory usage of httpd while you send some requests.
You may also want to strace the process:
# httpd -X &
# strace -o /tmp/strace.out -p `pidof httpd`
# tail -f /tmp/strace.out
Then hit the server with some requests and watch the strace output.
This is especially useful for finding I/O or IPC bottlenecks.
good luck
Tony
Re: maintaining shared memory size
Posted by Badai Aqrandista <ba...@hotmail.com>.
> > POE sounds good to start with, right?
>
>For a daemon to monitor a queue? Probably not. I'd just write a simple
>script that polls a database table and forks a new process to handle any
>jobs it finds. Parallel::ForkManager is handy for this kind of thing.
Yes. POE looks overkill compare to Parallel::ForkManager.
Thanks.
---
Badai Aqrandista
Cheepy (?)
_________________________________________________________________
Low rate ANZ MasterCard. Apply now!
http://clk.atdmt.com/MAU/go/msnnkanz0030000006mau/direct/01/ Must be over
18 years.
Re: maintaining shared memory size
Posted by Perrin Harkins <pe...@elem.com>.
On Wed, 2005-08-31 at 14:13 +1000, Badai Aqrandista wrote:
> Lots of SQL queries and calculation based on the query results. I've cached
> most of the query results, but the calculation results can't be cached
> because they depend on user inputs.
If this is serious number-crunching, it might be one of the few times
where using C really does make sense for that part.
> POE sounds good to start with, right?
For a daemon to monitor a queue? Probably not. I'd just write a simple
script that polls a database table and forks a new process to handle any
jobs it finds. Parallel::ForkManager is handy for this kind of thing.
- Perrin
Re: maintaining shared memory size
Posted by Badai Aqrandista <ba...@hotmail.com>.
>
>Okay, so it's slow because it does a lot of work. How does it do this
>search? Is it a bunch of SQL queries, or a request to a remote server of
>some other kind, or an internal calculation? Is there some possible
>caching you can do?
Lots of SQL queries and calculation based on the query results. I've cached
most of the query results, but the calculation results can't be cached
because they depend on user inputs.
>I would probably use some sort of queue system here. I would have the
>mod_perl process push requests onto a queue (probably in a database) and
>then have a separate daemon, maybe on a different machine, do the work and
>return the answer. It would allow you to split the search into multiple
>pieces that happen in parallel, and handle them on multiple machines
>simultaneously. It would also separate the web request, which is fast,
>from the slow search request. That lets your server scale to handling many
>more clients at once, even though some of them are just waiting for their
>turn.
Good idea!!! Thanks... You've been a great help...
POE sounds good to start with, right?
---
Badai Aqrandista
Cheepy (?)
_________________________________________________________________
Dating? Try Lavalife get 7 days FREE! Sign up NOW.
http://www.lavalife.com/clickthru/clickthru.act?id=ninemsn&context=an99&a=20233&locale=en_AU&_t=33473
Re: maintaining shared memory size
Posted by Perrin Harkins <pe...@elem.com>.
Badai Aqrandista wrote:
> My app is an online hotel booking system. That part is the part where it
> searches availabilities through all combination of dates, number of
> guests, rooms, packages, allotments, pricing structures, and more.
Okay, so it's slow because it does a lot of work. How does it do this
search? Is it a bunch of SQL queries, or a request to a remote server
of some other kind, or an internal calculation? Is there some possible
caching you can do?
> I must admit there is too much to handle in one mod_perl process. But I
> don't know any better way to design and code it. In the future, I'll
> probably move that search feature to a new process, so all the
> combinations can be calculated in parallel.
I would probably use some sort of queue system here. I would have the
mod_perl process push requests onto a queue (probably in a database) and
then have a separate daemon, maybe on a different machine, do the work
and return the answer. It would allow you to split the search into
multiple pieces that happen in parallel, and handle them on multiple
machines simultaneously. It would also separate the web request, which
is fast, from the slow search request. That lets your server scale to
handling many more clients at once, even though some of them are just
waiting for their turn.
For a simpler version that just forks and does the search on the same
machine, see this article by Randal:
http://www.stonehenge.com/merlyn/LinuxMag/col39.html
- Perrin
Re: maintaining shared memory size
Posted by Badai Aqrandista <ba...@hotmail.com>.
>>My app is an online hotel booking system. That part is the part where it
>>searches availabilities through all combination of dates, number of
>>guests, rooms, packages, allotments, pricing structures, and more.
>>
>This doesn't sound like a "search" at all. Typically individuals know at
>least SOMETHING about the
>booking they want to make. How many times do you book a hotel, but don't
>care whether it's in
>Barbados or Brazil?
Yes, the user provides the dates and the number of guests (and probably a
coupon number). Then based on that, the system searches for available room
and package combinations on the particular hotel, based on various
constraints.
---
Badai Aqrandista
Cheepy (?)
_________________________________________________________________
REALESTATE: biggest buy/rent/share listings
http://ninemsn.realestate.com.au
Re: maintaining shared memory size
Posted by colin_e <co...@bigfoot.com>.
Badai Aqrandista wrote:
> ...Your requests take 110 seconds each? What is your application doing?
> My app is an online hotel booking system. That part is the part where
> it searches availabilities through all combination of dates, number of
> guests, rooms, packages, allotments, pricing structures, and more.
>
This doesn't sound like a "search" at all. Typically individuals know at
least SOMETHING about the
booking they want to make. How many times do you book a hotel, but don't
care whether it's in
Barbados or Brazil?
If you have a resonable set of search criteria (constraints) linked to
appropriate indexes on your data,
you should be looking at <1s responses, not hundreds of seconds.
Many online systems will either reject a search with too few critieria,
or limit the results to (say) the
first 100 hits, precisely to stop silly or malicious searches producing
a zillion results and crippling their system.
If you really are enumerating all possible combinations, not really
searching for specific solutions, this
sounds like a job for an occasionally-run batch job storing the results
into a cache table, although in
your case this sounds like a potantially huge cache
Regards: Colin.
Re: maintaining shared memory size (was: Re:swampedwithconnection?)
Posted by Badai Aqrandista <ba...@hotmail.com>.
Hi,
>Rewriting things in C is a last resort. If you must do it, try
>Inline::C.
Yes... Inline::C looks good...
>Your requests take 110 seconds each? What is your application doing?
My app is an online hotel booking system. That part is the part where it
searches availabilities through all combination of dates, number of guests,
rooms, packages, allotments, pricing structures, and more.
I must admit there is too much to handle in one mod_perl process. But I
don't know any better way to design and code it. In the future, I'll
probably move that search feature to a new process, so all the combinations
can be calculated in parallel.
Thanks for helping...
---
Badai Aqrandista
Cheepy (?)
_________________________________________________________________
SEEK: Over 80,000 jobs across all industries at Australia's #1 job site.
http://ninemsn.seek.com.au?hotmail
Re: maintaining shared memory size (was: Re:
swampedwithconnection?)
Posted by Perrin Harkins <pe...@elem.com>.
On Tue, 2005-08-30 at 14:25 +1000, Badai Aqrandista wrote:
> I have localized one subroutine that has been heavily called in the search
> function. That should be the way to pick the candidate, shouldn't it?
What usually matters when working on speed is where the most wall clock
time is being spent. However, rewriting sections in C gives the most
benefit on subs that use the most CPU time rather than wall time.
> I have never done this and I am worried that writing it in C would push the
> project's deadline a little bit further.
It probably will. It will also make your code harder to maintain,
possibly introduce lots of new bugs, and maybe not help very much.
Rewriting things in C is a last resort. If you must do it, try
Inline::C.
> Anyhow, I've decrease the MaxClients to 3 and it worked!!! The speed
> significantly increases, from average of 110 sec/request to 85 sec/request.
> But, that's not enough. I need less than 10 sec/request.
Your requests take 110 seconds each? What is your application doing?
That doesn't sound like something that could be cured by simple
optimization.
- Perrin
Re: maintaining shared memory size (was: Re: swampedwithconnection?)
Posted by Badai Aqrandista <ba...@hotmail.com>.
>
>Then how do you know what to write in C?
>
I have localized one subroutine that has been heavily called in the search
function. That should be the way to pick the candidate, shouldn't it?
I have never done this and I am worried that writing it in C would push the
project's deadline a little bit further. However, does anyone have any hint
on doing this? What are the recommended readings?
Anyhow, I've decrease the MaxClients to 3 and it worked!!! The speed
significantly increases, from average of 110 sec/request to 85 sec/request.
But, that's not enough. I need less than 10 sec/request.
Is that possible without changing hardware?
---
Badai Aqrandista
Cheepy (?)
_________________________________________________________________
Sell your car for $9 on carpoint.com.au
http://www.carpoint.com.au/sellyourcar
Re: maintaining shared memory size (was: Re: swampedwithconnection?)
Posted by Badai Aqrandista <ba...@hotmail.com>.
>>
>>>You *do* have KeepAlive off in your httpd, right?
>>>
>>No...
>
>I mean in the backend Apache, not the frontend whatever.
Yes, I understand... I put it in the backend...
>When you're happering your server, is the CPU on the server
>running at or near 100%? If not, you have other problems.
Almost 100%, around 90%...
---
Badai Aqrandista
Cheepy (?)
_________________________________________________________________
Your opinion counts..for your chance to win a Mini Cooper click here
http://www.qualifiedopinions.com/joinup.php?source=hotmail
Re: maintaining shared memory size (was: Re: swampedwithconnection?)
Posted by David Hodgkinson <da...@gmail.com>.
On 30 Aug 2005, at 01:56, Badai Aqrandista wrote:
>
>
>> You *do* have KeepAlive off in your httpd, right?
>>
> No...
I mean in the backend Apache, not the frontend whatever.
When you're happering your server, is the CPU on the server
running at or near 100%? If not, you have other problems.
Re: maintaining shared memory size (was: Re: swampedwithconnection?)
Posted by Badai Aqrandista <ba...@hotmail.com>.
>
>On 29 Aug 2005, at 01:15, Badai Aqrandista wrote:
>>
>>I think I have to write some of the code in C. I can't find any other
>>places in the code to optimize (or probably I uncounciously don't want to
>>make changes because I don't have any test suites)...
>
>Then how do you know what to write in C?
The part that is called the most... I have localized it to one or two
subroutines... But I haven't done so... My C skill is very2 rusty...
>Franky, if your code is CPU bound (rather than waiting for the
>database) then a MaxClients of 3 will serve your purposes.
I'll give it a try...
>You *do* have KeepAlive off in your httpd, right?
No...
Thanks a lot for the enlightment...
---
Badai Aqrandista
Cheepy (?)
_________________________________________________________________
SEEK: Over 80,000 jobs across all industries at Australia's #1 job site.
http://ninemsn.seek.com.au?hotmail
Re: maintaining shared memory size (was: Re: swampedwithconnection?)
Posted by David Hodgkinson <da...@gmail.com>.
On 29 Aug 2005, at 01:15, Badai Aqrandista wrote:
>
> I think I have to write some of the code in C. I can't find any
> other places in the code to optimize (or probably I uncounciously
> don't want to make changes because I don't have any test suites)...
Then how do you know what to write in C?
Read and understand the chapter(s) in the mod_perl guide on
profiling and see where that takes you.
Franky, if your code is CPU bound (rather than waiting for the
database) then a MaxClients of 3 will serve your purposes.
You *do* have KeepAlive off in your httpd, right?
Re: maintaining shared memory size (was: Re: swampedwithconnection?)
Posted by Badai Aqrandista <ba...@hotmail.com>.
>>
>>top - 17:24:27 up 34 days, 9:01, 4 users, load average: 20.67, 12.84,
>>9.26
>>Tasks: 142 total, 7 running, 135 sleeping, 0 stopped, 0 zombie
>>Cpu(s): 88.7% us, 7.6% sy, 0.0% ni, 0.0% id, 2.0% wa, 0.0% hi, 1.7%
>>si
>>Mem: 906736k total, 359464k used, 547272k free, 6184k buffers
>>Swap: 3036232k total, 111564k used, 2924668k free, 17420k cached
>
>Which processes are swapping?
>
>Have you tried setting MaxClients to say 10 (or some other low number
>that'll ensure you don't run out of memory).
>
I just did, and cleared the swap before doing another testing. The result:
Nothing swapped, but the average response time goes down by 20 seconds. I
guess Perrin was right, with 30 MaxClients, the processes are competing the
CPU cycles.
I think I have to write some of the code in C. I can't find any other places
in the code to optimize (or probably I uncounciously don't want to make
changes because I don't have any test suites)...
---
Badai Aqrandista
Cheepy (?)
_________________________________________________________________
Low rate ANZ MasterCard. Apply now!
http://clk.atdmt.com/MAU/go/msnnkanz0030000006mau/direct/01/ Must be over
18 years.
Re: maintaining shared memory size (was: Re: swampedwithconnection?)
Posted by Ask Bjørn Hansen <as...@develooper.com>.
On Aug 24, 2005, at 0:49, Badai Aqrandista wrote:
> I have put a reverse procy in front of my mod_perl servers and I
> have set MaxClient to 30. I have tried setting it to 50, but it
> slows down the response time.
>
> This is what top gave me when I hammered the test server with httperf:
>
> ----------
>
> top - 17:24:27 up 34 days, 9:01, 4 users, load average: 20.67,
> 12.84, 9.26
> Tasks: 142 total, 7 running, 135 sleeping, 0 stopped, 0 zombie
> Cpu(s): 88.7% us, 7.6% sy, 0.0% ni, 0.0% id, 2.0% wa, 0.0%
> hi, 1.7% si
> Mem: 906736k total, 359464k used, 547272k free, 6184k
> buffers
> Swap: 3036232k total, 111564k used, 2924668k free, 17420k
> cached
Which processes are swapping?
Have you tried setting MaxClients to say 10 (or some other low number
that'll ensure you don't run out of memory).
- ask
--
http://www.askbjoernhansen.com/
Re: maintaining shared memory size (was: Re:swampedwithconnection?)
Posted by Badai Aqrandista <ba...@hotmail.com>.
Hi all,
>If the performance you get is not enough for you, look for the
>bottleneck. This could be running out of CPU, running out of memory, or
>contention for a shared resource like a database. Then you fix it, by
>changing code or buying hardware.
I'll try that...
Thanks for helping me out... Especially perrin... I'll let you know how it
goes...
---
Badai Aqrandista
Cheepy (?)
_________________________________________________________________
SEEK: Over 80,000 jobs across all industries at Australia's #1 job site.
http://ninemsn.seek.com.au?hotmail
Re: maintaining shared memory size (was: Re:swampedwithconnection?)
Posted by Perrin Harkins <pe...@elem.com>.
On Thu, 2005-08-25 at 17:46 +1000, Badai Aqrandista wrote:
> To reduce memory consumption -> look for ways to do more things on compile
> time
Also avoid loading large things into memory all at once (e.g. slurping
files), pass references instead of copies, use the disk more, etc.
> To reduce CPU usage -> write the most called subroutine in C
More likely would be to look at the approach you used in the slow code
(which you find with profiling) and see if you can change the algorithm
to something that does less work. That usually gives the largest gains.
Sometimes you may be using a CPAN module (LWP, Cache::Cache) which has a
faster alternative. Sometimes caching objects (DBI statement handles
for example) helps.
> To reduce database contention -> cache as much as possible
That can help with slow read-only queries. For actual contention (which
implies writes and locks) you usually need to either change your access
patterns or use a different database (InnoDB tables in MySQL, Postgres,
etc.)
- Perrin
Re: maintaining shared memory size (was: Re:swampedwithconnection?)
Posted by Badai Aqrandista <ba...@hotmail.com>.
>If the performance you get is not enough for you, look for the
>bottleneck. This could be running out of CPU, running out of memory, or
>contention for a shared resource like a database. Then you fix it, by
>changing code or buying hardware.
I think I've got the point now:
To reduce memory consumption -> look for ways to do more things on compile
time
To reduce CPU usage -> write the most called subroutine in C
To reduce database contention -> cache as much as possible
Thanks for all your help...
The journey just begins...
---
Badai Aqrandista
Cheepy (?)
_________________________________________________________________
Dating? Try Lavalife get 7 days FREE! Sign up NOW.
http://www.lavalife.com/clickthru/clickthru.act?id=ninemsn&context=an99&a=20233&locale=en_AU&_t=33473
Re: maintaining shared memory size (was: Re:
swampedwithconnection?)
Posted by Perrin Harkins <pe...@elem.com>.
On Wed, 2005-08-24 at 17:49 +1000, Badai Aqrandista wrote:
> I have put a reverse procy in front of my mod_perl servers and I have set
> MaxClient to 30. I have tried setting it to 50, but it slows down the
> response time.
Are you running out of memory when you set it to 50? That's what you
are trying to avoid by setting MaxClients. Also, you are sending your
load test through the proxy server, and that has a much higher
MaxClients setting, right?
> I ran httperf to create 50 connections. This is the result:
>
> 1 connections per second -> avg reply time = 103953.7 ms
> 10 connections per second -> avg reply time = 123167.2 ms
> 20 connections per second -> avg reply time = 121483.7 ms
> 30 connections per second -> avg reply time = 114411.3 ms
> 40 connections per second -> avg reply time = 130168.7 ms
> 50 connections per second -> avg reply time = 130457.4 ms
That looks pretty good to me. It scales better than linearly, i.e 50
conns is not 50 times slower than 1 conn.
> When only creating 1 connection, the avg reply time = 3289.4 ms
That doesn't seem to be what your numbers above say.
> I have no idea where to start fixing this. It seems that the more connection
> there is, the higher the avg reply time.
That's true for any web server. More connections means the machine is
doing more work and running more processes, so of course it will be
slower. All you need to do is make sure you are not running out of
memory and going into swap.
If the performance you get is not enough for you, look for the
bottleneck. This could be running out of CPU, running out of memory, or
contention for a shared resource like a database. Then you fix it, by
changing code or buying hardware.
- Perrin
Re: maintaining shared memory size (was: Re: swampedwithconnection?)
Posted by Badai Aqrandista <ba...@hotmail.com>.
> > Does this sound like fixing the wrong problem?
>
>Yes. Put a reverse proxy in front of your server, tune MaxClients so
>you won't go into swap, and then benchmark to see how much load you can
>handle. Then think about tuning.
Thanks for replying...
I have put a reverse procy in front of my mod_perl servers and I have set
MaxClient to 30. I have tried setting it to 50, but it slows down the
response time.
This is what top gave me when I hammered the test server with httperf:
----------
top - 17:24:27 up 34 days, 9:01, 4 users, load average: 20.67, 12.84,
9.26
Tasks: 142 total, 7 running, 135 sleeping, 0 stopped, 0 zombie
Cpu(s): 88.7% us, 7.6% sy, 0.0% ni, 0.0% id, 2.0% wa, 0.0% hi, 1.7% si
Mem: 906736k total, 359464k used, 547272k free, 6184k buffers
Swap: 3036232k total, 111564k used, 2924668k free, 17420k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
22802 www-data 15 0 39664 35m 7968 S 30.7 4.0 0:05.02 apache-perl
22370 www-data 16 0 39664 35m 7968 S 12.8 4.0 0:03.45 apache-perl
13604 www-data 17 0 40096 35m 7968 R 5.7 4.0 0:30.93 apache-perl
15424 root 15 0 32060 7668 1560 S 3.8 0.8 6:49.47 memcached
13611 www-data 15 0 40036 35m 7968 S 3.5 4.0 0:17.13 apache-perl
22804 www-data 17 0 39664 35m 7968 R 3.1 4.0 0:03.01 apache-perl
13612 www-data 16 0 40032 35m 7968 S 0.7 4.0 0:33.38 apache-perl
------------
I ran httperf to create 50 connections. This is the result:
1 connections per second -> avg reply time = 103953.7 ms
10 connections per second -> avg reply time = 123167.2 ms
20 connections per second -> avg reply time = 121483.7 ms
30 connections per second -> avg reply time = 114411.3 ms
40 connections per second -> avg reply time = 130168.7 ms
50 connections per second -> avg reply time = 130457.4 ms
When only creating 1 connection, the avg reply time = 3289.4 ms
When creating 10 connections, with 1 conn per second, the avg reply time =
25929.7 ms
I have no idea where to start fixing this. It seems that the more connection
there is, the higher the avg reply time.
Does anyone ever experienced this? Why does it happen?
Basically I just want to make my webapp ready to launch... (doesn't anyone?)
THANK YOU...
---
Badai Aqrandista
Cheepy (?)
_________________________________________________________________
SEEK: Over 80,000 jobs across all industries at Australia's #1 job site.
http://ninemsn.seek.com.au?hotmail
Re: maintaining shared memory size (was: Re: swamped
withconnection?)
Posted by Perrin Harkins <pe...@elem.com>.
On Wed, 2005-08-24 at 10:31 +1000, Badai Aqrandista wrote:
> Anyway, to fix this, I'm trying to make my onw shared memory with
> Apache::SharedMem.
Don't use that module. It's very inefficient.
> But it seems that shared memory is just a memory area
> that any process can read from or write to, not a memory that can become
> part of any process's memory space.
>
> I'd like to put the read only data in a memory area that can become part of
> any process's memory space, like threads sharing memory space. Is there any
> way I can do that?
You can share read-only data by loading it into normal variables during
startup. You can share read/write data by keeping it in a database
(including things like BerkeleyDB or Cache::FastMmap) and only reading
the small parts of it that you need in each process.
There is no way to access data in perl without the size of the process
you read the data from allocating memory to hold that data while you use
it. In other words, shared memory is not useful for reducing the size
of existing processes unless those processes are currently each holding
a lot of data that they never use.
> Does this sound like fixing the wrong problem?
Yes. Put a reverse proxy in front of your server, tune MaxClients so
you won't go into swap, and then benchmark to see how much load you can
handle. Then think about tuning.
- Perrin
Re: maintaining shared memory size (was: Re: swamped withconnection?)
Posted by Badai Aqrandista <ba...@hotmail.com>.
>On Tue, 2005-08-23 at 17:23 +1000, Badai Aqrandista wrote:
> > How do I maintain the size of the shared memory between apache children?
> > What cause a memory page to be copied (not shared) from perl's point of
> > view?
>
>Anything that writes to memory -- modifying any variable (even just
>reading one in a different context) or compiling some code are the most
>common things. There's a bit more here:
>http://modperlbook.com/html/ch10_01.html
Oh my, I should've kept memory consumption in mind when designing the code
in the first place.
Anyway, to fix this, I'm trying to make my onw shared memory with
Apache::SharedMem. But it seems that shared memory is just a memory area
that any process can read from or write to, not a memory that can become
part of any process's memory space.
I'd like to put the read only data in a memory area that can become part of
any process's memory space, like threads sharing memory space. Is there any
way I can do that? How about mmap? I have never looked into that.
Does this sound like fixing the wrong problem?
Thank you...
---
Badai Aqrandista
Cheepy (?)
_________________________________________________________________
On the road to retirement? Check out MSN Life Events for advice on how to
get there! http://lifeevents.msn.com/category.aspx?cid=Retirement
Re: maintaining shared memory size (was: Re: swamped with
connection?)
Posted by Perrin Harkins <pe...@elem.com>.
On Tue, 2005-08-23 at 17:23 +1000, Badai Aqrandista wrote:
> How do I maintain the size of the shared memory between apache children?
> What cause a memory page to be copied (not shared) from perl's point of
> view?
Anything that writes to memory -- modifying any variable (even just
reading one in a different context) or compiling some code are the most
common things. There's a bit more here:
http://modperlbook.com/html/ch10_01.html
- Perrin
maintaining shared memory size (was: Re: swamped with connection?)
Posted by Badai Aqrandista <ba...@hotmail.com>.
I think I have to reword the question:
How do I maintain the size of the shared memory between apache children?
What cause a memory page to be copied (not shared) from perl's point of
view?
>
>This brings the question of how to increase shared memory usage. I've tried
>to load every modules upfront. But even before any request comes in, the
>shared memory is only 7 Mb. What makes it so small?
>
Thanks...
---
Badai Aqrandista
Cheepy (?)
_________________________________________________________________
Your opinion counts..for your chance to win a Mini Cooper click here
http://www.qualifiedopinions.com/joinup.php?source=hotmail
Re: swamped with connection?
Posted by Badai Aqrandista <ba...@hotmail.com>.
>Did you run out of memory? Is the CPU pegged? Give us something to go
>on here...
>Have you read the performance tuning docs on the mod_perl site?
Yes, ages ago. I just read it again and did the calculation again.
Apparently, yes, it runs out of memory and holds up the connection so no
other requests can be processed.
So, here's the variables:
RAM = 700 Mb
Per process total size = 40 Mb
Shared memory = 7 Mb
So, the number of processes = (700 - 7) / 33 = 21 processes
So, does that mean it can only accept up to 21 connections?
This brings the question of how to increase shared memory usage. I've tried
to load every modules upfront. But even before any request comes in, the
shared memory is only 7 Mb. What makes it so small?
Thank you...
---
Badai Aqrandista
Cheepy (?)
_________________________________________________________________
REALESTATE: biggest buy/rent/share listings
http://ninemsn.realestate.com.au
Re: swamped with connection?
Posted by Perrin Harkins <pe...@elem.com>.
On Tue, 2005-08-23 at 10:20 +1000, Badai Aqrandista wrote:
> I am still trying to improve my company's webapp performance. I'm testing it
> with httperf and autobench. The application seems to be able to respond when
> hammered by 20 connections per second and 10 calls per connection. But then,
> it doesn't respond to any request when the connection rate is raised to 40
> (with 10 calls per connection) and above.
Did you run out of memory? Is the CPU pegged? Give us something to go
on here...
> What does the apache treats as a request (hence forks another child)?
> - every new connection, or
> - every http request regardless of it comes with a new connection or comes
> through an existing connection
Every new connection, if you use Keep-Alive or HTTP 1.1.
> My only guess is because the connections goes over the MaxClients count
> while the existing apache children are still processing the previous
> requests.
What is your MaxClients set to?
Have you read the performance tuning docs on the mod_perl site?
- Perrin