[users@httpd] memory problem with apache on Linux

[Malka Cymbalista <Ma...@weizmann.ac.il>]

We are running apache 2.2.6  with mod_perl 2.0.3 on  a 64bit Linux machine.
We have recently been experiencing some very strange problems.  Every now and then (and we cannot find any consistency), the server slows down to a crawl and gets stuck for a while (it can be up to 10 minutes) and then something frees up and everythin goes back to normal. 
 
I have kept the top command open for the past several hours and I can see the when the server gets stuck, there is almost no free memory.  It often happens that there is very little memory but all the processes in top have 0.0% memory usage.  Every now and then, an httpd process will show up that can climb as high as 95% and then we see the memory going down, the machine may get stuck, and then the process will disappear from top and the memory will go way up and all will be well. 
 
We noticed the following that may or may not be connected:
1. The following line appears many times in the apache access log:
::1 - - [08/Jul/2008:15:46:30 +0300] "GET / HTTP/1.0" 200 55897 "-"  "Apache/2.2.6 (Unix) mod_ssl/2.2.6 \xf8\xdd\xab PHP/5.2.4 mod_perl/2.0.3 Perl/v5.8.5 (internal dummy connection)"
2. Around the time that the machine gets stuck we see the following im our messages file:
Out of Memory: Killed process 19182 (httpd).
3. At around the time that the machine was stuck nothing was written to the apache access log for about 8 minutes.
 
Does anyone have any pointers as to where we can look for clues as to what is going on.



 
 
Malka Cymbalista
Webmaster, Weizmann Institute of Science
malki.cymbalista@weizmann.ac.il 
08-934-3036