general performance issues

[André Warnier <aw...@ice-sa.com>]

Hi.

This is not really an on-topic question, and is more of a general one.
But I figure that, for this type of "general", the combination of skills 
of the people on this list might be just right to provide some pointers.
If it's too off-topic nevertheless, just send me packing, I won't insist.

The "top" snapshot below shows one of our old clunkers, doing 
essentially nothing for now.  It's an average desktop PC from a few 
years back, running Linux Debian Etch, with a venerable Pentium II as 
processor, 512 Mb Ram and your average PC disks from back then.  It is 
so old that there is in fact a problem still finding the memory cards 
that fit in it.
Yet most of the time it faithfully does its job, which is to be a 
development and test machine for a small team of programmers.
But sometimes it doesn't, and becomes rather slow for a while.

The programs that run on it are fairly typical of our production 
machines (which are much newer/bigger/better), which is why I ask. I 
figure that considering the respectable age of this system and its 
limited everything, anything that we would do to improve its generic 
performance would be all the more effective if applied to the production 
servers, right ? or wrong ? in a generic way I mean.
Also, I want to add that we do not really have a problem yet, I am just 
trying to gather some general information and general impressions,maybe 
collect some tricks or tools for the future. We can easily get another 
more recent PC to replace that one. But maybe then it would be harder to 
put it under stress and see where there is a bottleneck.

We thus run on these machines : Apache 2.x with mod_perl 2.x, Tomcat 
5.5, and mod_jk 1.2.x to connect them.  Under Tomcat run a couple of 
very small servlets, and one that seems to be really huge compared to 
the others. Under Apache directly, we have a series of application 
components running as mod_perl modules or as cgi-bin scripts.
We always access our web applications through Apache.
Apart from Tomcat and Apache-based stuff, there are also a few other
"daemons" running which you can see.  A couple of them (the next 2 after 
Tomcat) are java daemons,
and they process (and parse) XML files all day.  A couple of stand-alone 
perl processes are also daemons, and also process XML data.
The two "star" processes at the bottom are parts of a special database 
system, written in C.
The one "python" process running there is "denyhosts".

This "top" display is sorted by "memory" (one of the options of memory 
in top, "n", I'm not quite sure what it corresponds to).
The first thing that strikes me, is the huge memory allocated to Tomcat 
and the other Java processes.  I wonder if this unavoidable, or if it is 
just misconfiguration on my part.
The "huge Tomcat application" is unfortunately one that we just buy, we 
can't peek in the code to see what's going on there.  We just know it is 
(also) processing some large quantity of XML data, mostly when it starts 
up.  That almost paralyses our server for 10 minutes or so when we 
restart Tomcat.
The other question is, does the memory shown as used by the Apache 
processes seem normal/abnormal to anyone ?  This being a prefork Apache, 
if it seems abnormal, what kind of thing should I look at ?  Or is it so 
normal that I should not even spend time on it ?
(I already load a fair number of modules in a startup script, but maybe 
I should load even more of them there, if there is no penalty).

Thanks for any reactions, even if it is to tell me that I don't know 
what I'm talking about or that this isn't the right place. But then, be 
as kind maybe as pointing me to the right place.


Sample "top" :
   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
  6226 tomcat55  25   0  506m 188m  10m S  0.0 37.3 127:01.74 java
  5937 root      15   0  267m  28m 8056 S  0.0  5.6  24:20.96 java
  8613 root      24   0  260m  15m 9332 S  0.0  3.1   0:02.16 java
  8640 root      15   0 29560  15m 5984 S  0.0  3.0   0:02.10 apache2
  8632 efs       15   0 16964  12m 1968 S  0.0  2.6   0:06.96 MiraLoader.pl
  8665 www-data  15   0 29692  12m 3016 S  0.0  2.4   0:08.82 apache2
  8662 www-data  15   0 29692  12m 3016 S  0.0  2.4   0:08.74 apache2
12414 www-data  15   0 29560  10m 1624 S  0.0  2.2   0:08.32 apache2
12415 www-data  15   0 29560  10m 1624 S  0.0  2.2   0:08.33 apache2
18880 www-data  15   0 29560  10m 1624 S  0.0  2.2   0:04.84 apache2
18881 www-data  18   0 29560  10m 1624 S  0.0  2.2   0:04.37 apache2
18882 www-data  15   0 29560  10m 1624 S  0.0  2.2   0:04.55 apache2
  8666 efs       15   0 13532 9724 2000 S  0.0  1.9   0:02.40 MiraFiler.pl
21408 root      18   0  9940 3884 1376 S  0.0  0.8  32:10.91 python
24934 efs       17   0 11708 2812 2172 S  0.0  0.5   0:00.06 smbd
23831 efs       18   0 14548 2680 1724 S  0.0  0.5   0:03.35 MiraASci.pl
23848 efs       15   0 11696 2568 1708 S  0.0  0.5   0:03.03 ifsser.pl
24752 root      15   0  7804 2324 1836 S  0.0  0.4   0:00.12 sshd
24763 root      15   0  7840 2292 1848 S  0.3  0.4   0:00.32 sshd
26537 root      16   0 11416 2104 1624 S  0.0  0.4   3:57.05 smbd
27888 star      15   0  6600 1968 1544 S  0.0  0.4   0:03.86 star
27886 star      16   0  5004 1840 1440 S  0.0  0.4   0:30.21 star

André

Re: general performance issues

[John ORourke <jo...@o-rourke.org>]

André Warnier wrote:
> If it's too off-topic nevertheless, just send me packing, I won't insist.

I might be alone in this but I think apache memory use is something all 
heavy mod_perl users have to get to grips with, so I mentally 
s/java|tomcat/mod_perl/g'd and read your very long email....
>
> But sometimes it doesn't, and becomes rather slow for a while.
Find out if it's *actual* slowness or just HTTP requests being queued up 
because none of the apache children are free to service the request.
>
> anything that we would do to improve its generic performance would be 
> all the more effective if applied to the production servers, right ? 
> or wrong ? in a generic way I mean.
I agree but with one caveat - memory is very cheap at the moment.  
Strike a good balance between spending hours trimming the memory 
footprint of your app and buying more RAM.

> The first thing that strikes me, is the huge memory allocated to 
> Tomcat and the other Java processes.  I wonder if this unavoidable, or 
> if it is just misconfiguration on my part.
> The "huge Tomcat application" is unfortunately one that we just buy, 
> we can't peek in the code to see what's going on there.  We just know 
> it is (also) processing some large quantity of XML data, mostly when 
> it starts up.  That almost paralyses our server for 10 minutes or so 
> when we restart Tomcat.
Tongue in cheek, thats what you get for not rewriting it in perl ;)  
Seriously though, in your own apps make sure you're not loading large 
XML files fully into RAM before parsing them.  Use a SAX or similar 
parser to treat large XML files like a stream.
> The other question is, does the memory shown as used by the Apache 
> processes seem normal/abnormal to anyone ?  This being a prefork 
> Apache, if it seems abnormal, what kind of thing should I look at ?  
> Or is it so normal that I should not even spend time on it ?
Those are very lean looking apache's by my own experience, I don't think 
you'll do much better.

hope that's useful,

John