mod_proxy + mod_mem_cache memory consumption issues

[Manik Taneja <mt...@movik.net>]

Hi,

I have a ubuntu server with a  linux 2.6.24 kernel running on an HP  
proliant server with 64-bit 2  x dual-core, 4 GB ram.   I'm trying to  
measure the peak request rate with the apache server configured as  
forward proxy and so I load this system with 5000 req/sec from a  
polygraph setup.    I have configured mod_mem_cache to only use about  
2GB and the max_object _count setting is also configured such that it  
is approximately  = "cache size / mean size of the object".  Now I  
have a added log messages to the mod_mem_cache code, so that when the  
cache limit is reached a log messages will be dumped and from then on  
the replacement algorithm can take over. The polygraph server has been  
configured to offer an object-hit ratio of about 30%.

Since, the mem_cache doesn't use shared-memory I use a single apache  
server process, i.e MaxServers is set to 1. If i used more than one  
apache process then i wouldn't be able to control which requests go to  
which apache server and therefore it is a strong possibility that i  
might end up caching the same object multiple times each in the  
context of an individual apache child process. I am using the event  
MPM, because I thought that would give me better performance than the  
prefork and worker MPMs.  I'm aware that the event MPM is an  
experimental module but for achieving the best server performance is  
far important than a few bugs that we might encounter on the way.

Anyway, when i examine the memory consumption of the httpd process via  
the top command, I see that the process has consumed up to 3.7 GB of  
heap space even though the maximum size of the cache has been set to  
2GB. At this point the server has already run for about 30 minutes  
serving 5k req/sec. I initially thought that this might be because of  
the back-log requests and the apr_pool_ memory allocations so in the  
next run I completely disabled the mem_cache module.  During this run,  
I saw that apache was only able to serve around 4000 req/sec, because  
all the requests were cache misses however, the memory footprint of  
apache after 45 minutes was only around 40mb.  As soon as  i enable  
the cache, I observe that the memory footprint of apache is almost 0.5  
- 1 .5 times as that of the current cache size.  The cache here hasn't  
even reached its limit so there is no possibility of ejected objects  
not being freed.

I found this link that touches on this topic and it suggests that the  
apache worker processes  should be restarted via the  
MaxRequestsPerChild directive in order to avoid problems due to memory  
fragmentation and leaks. (http://www.apachesecurity.net/blog/2006/08/apache_reverse_proxy_memory_co.html 
).  Restarting the child process is not an option for me since I will  
lose all the objects that I have cached. If the problem is due to  
memory fragmentation because the cached object sizes tend to vary  
between 3k - 8k in my test, then that too is also something that I can  
live with. I could write my own memory allocation module that  
allocates chunks in slabs so that way memory consumption is  
predictable.  The only apr_pool that the mem_cache seems to use is  
when it allocates a cache_object and I don't see that as a reason for  
this large memory consumption. I am beginning to believe that this  
indeed might be a memory fragmentation issue, however its only been a  
week since I have been working with the apache code for a week and so  
I am not entire sure about my hypothesis.

I would really appreciate if someone on this list could communicate  
what their thoughts on this matter and if you have reached this far  
then thank you for your time.

Best Regards,
Manik

Re: mod_proxy + mod_mem_cache memory consumption issues

[Manik Taneja <mt...@movik.net>]

thats not a bad idea except for that fact that mod_disk_cache lacks a  
replacement algorithm.  As per my understanding it relies on  
HTcacheclean to keep the cache size in check.

i suppose this might be worth trying out though.

Thanks,
Manik

On 18-Sep-08, at 7:05 PM, Akins, Brian wrote:

> Have you tried mod_disk_cache with the cachroot in /dev/shm?   
> Mod_mem_cache,
> it seems, is negleted in comparison to the disk one.  Using /dev/shm  
> is
> basically putting the cache in shared emmory.
>
> -- 
> Brian Akins
> Chief Operations Engineer
> Turner Digital Media Technologies
>

Re: mod_proxy + mod_mem_cache memory consumption issues

["Akins, Brian" <Br...@turner.com>]

Have you tried mod_disk_cache with the cachroot in /dev/shm?  Mod_mem_cache,
it seems, is negleted in comparison to the disk one.  Using /dev/shm is
basically putting the cache in shared emmory.

-- 
Brian Akins
Chief Operations Engineer
Turner Digital Media Technologies