You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by Alex Varju <va...@antiflux.org> on 2002/11/28 03:42:19 UTC
Memory leak on Windows [Bugzilla #11427]
Hi there,
I am trying to track down the cause of this memory growth problem, but I'm
not really sure where to start.
In the hopes that I could at least figure out where the memory was first
allocated, I've tried running Apache under Purify. Unfortunately I can't
seem to get a debug version built that does not end up loading the
non-debug msvcrt.dll. Has anybody successfully run the Windows version
inside Purify? If so, are there any magic tricks that you could share
with me?
Thanks,
Alex
RE: Memory leak on Windows [Bugzilla #11427]
Posted by Cliff Woolley <jw...@virginia.edu>.
On Thu, 19 Dec 2002, Alex Varju wrote:
> apr_allocator_max_free_set(), and the default behaviour is to hold on to
> all memory ever allocated, and allocate memory future requests out of that
That's a general problem, yes... one for which I have a patch in my local
tree that I'll commit next week sometime when I get a chance to breathe.
:)
--Cliff
RE: Memory leak on Windows [Bugzilla #11427]
Posted by Alex Varju <va...@antiflux.org>.
I believe that I have made some progress on this, but I'm not confident
that my solution is the correct one. There were several factors playing
into this that pulled my attention away from the real issue.
I originally reported that this problem was CGI specific; it turns out
that this is not the case. The way I was reproducing what appeared to be
a major leak was by sending a number of parallel requests to CGIs that
generate a lot of output. The result of this was that Apache needed a lot
of temporary buffers at the same time. After digging around for a while,
I discovered that Apache on Windows never calls
apr_allocator_max_free_set(), and the default behaviour is to hold on to
all memory ever allocated, and allocate memory future requests out of that
block. When I spawned the parallel requests, Apache grabbed a whole bunch
of memory, and then kept reusing it. After setting a limit here, I was
able to move on to the bigger problem.
After making the change above, I realized that every request was leaking a
little bit, not just the CGI requests. To confirm this, I made a simple
configuration file that redirects all requests to a static HTML file.
Sure enough, it was leaking.
The problem, as best as I can tell, is that mpm_winnt calls
apr_bucket_alloc_create() once for each thread, and registers it in the
pchild pool. This bucket allocator is then passed through to
core_create_conn(), and used for all apr_bucket_XXX() routines during the
request. The pchild pool is not cleared until the server shuts down, so
the memory used here grows and grows. To solve this problem I changed
mpm_winnt so that it creates the bucket allocator using the ptrans pool,
which gets cleared after every connection is finished.
After making this change, the system behaved much better. Just to check,
I then undid my first change related to the maximum memory to hold on to,
and the system continued to function corrrectly. So in the end, I only
needed the one fix.
The following patch shows the changes I made. My question now for the
experts is whether this will break anything.
Thanks,
Alex.
Index: server/mpm/winnt/child.c
===================================================================
RCS file: /home/cvspublic/httpd-2.0/server/mpm/winnt/child.c,v
retrieving revision 1.9
diff -u -u -r1.9 child.c
--- server/mpm/winnt/child.c 14 Oct 2002 14:54:45 -0000 1.9
+++ server/mpm/winnt/child.c 20 Dec 2002 01:02:38 -0000
@@ -122,6 +122,7 @@
if (context) {
apr_pool_clear(context->ptrans);
context->next = NULL;
+ context->ba = apr_bucket_alloc_create(context->ptrans);
ResetEvent(context->Overlapped.hEvent);
apr_thread_mutex_lock(qlock);
if (qtail)
@@ -187,7 +188,7 @@
apr_pool_tag(context->ptrans, "ptrans");
context->accept_socket = INVALID_SOCKET;
- context->ba = apr_bucket_alloc_create(pchild);
+ context->ba = apr_bucket_alloc_create(context->ptrans);
apr_atomic_inc(&num_completion_contexts);
}
@@ -416,7 +417,7 @@
context = apr_pcalloc(pchild, sizeof(COMP_CONTEXT));
apr_pool_create(&context->ptrans, pchild);
apr_pool_tag(context->ptrans, "ptrans");
- context->ba = apr_bucket_alloc_create(pchild);
+ context->ba = apr_bucket_alloc_create(context->ptrans);
}
while (1) {
RE: Memory leak on Windows [Bugzilla #11427]
Posted by Alex Varju <va...@antiflux.org>.
Hi Bill,
Thank you for looking at this.
I've just grabbed the httpd-2.0, apr, apr-util, and apr-iconv modules from
CVS (-rHEAD). With the same configuration file as I was using in 2.0.43,
I can still reproduce the problem.
I then added 'KeepAlive off' to my configuration file. With this setting,
the problem continues to display itself on my server. With 'KeepAlive on'
and 'MaxKeepAliveRequests 10' in my configuration file, again the problem
continues.
I am running two different tests to reproduce this, although they may not
be related. In the first test, I have a client machine running
SilkPerformer which opens up several parallel requests to the server. In
the second test, I use Internet Explorer and hold down the F5 key for a
few seconds straight .. I think this triggers the client abort code inside
Apache.
Any other suggestions? My plan now is to walk through all the memory
allocations in the call stack in the hopes that I find something using a
pool that doesn't get released. I'm definitely not looking forward to
this.
Thanks,
Alex.
On Tue, 17 Dec 2002, Bill Stoddard wrote:
> I am running 2.0.44-dev (latest snapshot in CVS from just a few minutes ago) and
> I'm not able to recreate this problem. Have you tried the latest version of 2.0
> (CVS HEAD)? Just for fun, try disabling keep-alive connections or set
> MaxKeepAliveRequests to a small value (maybe from the default of 100 to 10) and
> report back the results.
>
> Bill
>
> >
> > Hi,
> >
> > I'm still trying to track the cause of this problem down, and I'm hoping
> > somebody around here can help me.
> >
> > To summarize, I'm seeing Apache's memory footprint grow abnormally large
> > on Windows when using CGIs. The size of the growth seems to be
> > proportional to the amount of data printed to stdout from the CGI.
> >
> > Some sample data:
> > - With 16 threads and 1 meg sent to stdout, combined physical and virtual
> > memory reaches about 70 megs after hammering the server for several
> > minutes
> > - If I increase the amount of stdout data to 2 megs, the process grows to
> > about 130 megs within another few minutes.
> >
> > I've spent the last few days reviewing the code, and I'm a bit confused
> > about the mpm_winnt pool cleanup code. I haven't spent a lot of time
> > reading the code in the past, so there's a good chance that I've just
> > missed something. As far as I can tell, ap_read_request() creates the
> > request pool, but nothing explicitly cleans it up. Instead, it looks like
> > mpm_recycle_completion_context() clears the ptrans pool the next time the
> > thread handles a request. While this seems funny to me, I don't see why
> > some of the memory would fail to be released.
> >
> > I've tried to simplify my httpd.conf file to reduce the test case:
> >
> > ServerRoot c:/varju/webct/webct/server
> > ThreadsPerChild 16
> > LoadModule cgi_module modules/mod_cgi.so
> > LoadModule alias_module modules/mod_alias.so
> > Listen 80
> > ScriptAlias /test.pl c:/varju/webct/webct/webct/generic/public/test.pl
> >
> > Can anybody offer me suggestions of where to look next?
> >
> > Thanks,
> > Alex.
>
>
RE: Memory leak on Windows [Bugzilla #11427]
Posted by Bill Stoddard <bi...@wstoddard.com>.
I am running 2.0.44-dev (latest snapshot in CVS from just a few minutes ago) and
I'm not able to recreate this problem. Have you tried the latest version of 2.0
(CVS HEAD)? Just for fun, try disabling keep-alive connections or set
MaxKeepAliveRequests to a small value (maybe from the default of 100 to 10) and
report back the results.
Bill
>
> Hi,
>
> I'm still trying to track the cause of this problem down, and I'm hoping
> somebody around here can help me.
>
> To summarize, I'm seeing Apache's memory footprint grow abnormally large
> on Windows when using CGIs. The size of the growth seems to be
> proportional to the amount of data printed to stdout from the CGI.
>
> Some sample data:
> - With 16 threads and 1 meg sent to stdout, combined physical and virtual
> memory reaches about 70 megs after hammering the server for several
> minutes
> - If I increase the amount of stdout data to 2 megs, the process grows to
> about 130 megs within another few minutes.
>
> I've spent the last few days reviewing the code, and I'm a bit confused
> about the mpm_winnt pool cleanup code. I haven't spent a lot of time
> reading the code in the past, so there's a good chance that I've just
> missed something. As far as I can tell, ap_read_request() creates the
> request pool, but nothing explicitly cleans it up. Instead, it looks like
> mpm_recycle_completion_context() clears the ptrans pool the next time the
> thread handles a request. While this seems funny to me, I don't see why
> some of the memory would fail to be released.
>
> I've tried to simplify my httpd.conf file to reduce the test case:
>
> ServerRoot c:/varju/webct/webct/server
> ThreadsPerChild 16
> LoadModule cgi_module modules/mod_cgi.so
> LoadModule alias_module modules/mod_alias.so
> Listen 80
> ScriptAlias /test.pl c:/varju/webct/webct/webct/generic/public/test.pl
>
> Can anybody offer me suggestions of where to look next?
>
> Thanks,
> Alex.
Re: Memory leak on Windows [Bugzilla #11427]
Posted by Alex Varju <va...@antiflux.org>.
Hi,
I'm still trying to track the cause of this problem down, and I'm hoping
somebody around here can help me.
To summarize, I'm seeing Apache's memory footprint grow abnormally large
on Windows when using CGIs. The size of the growth seems to be
proportional to the amount of data printed to stdout from the CGI.
Some sample data:
- With 16 threads and 1 meg sent to stdout, combined physical and virtual
memory reaches about 70 megs after hammering the server for several
minutes
- If I increase the amount of stdout data to 2 megs, the process grows to
about 130 megs within another few minutes.
I've spent the last few days reviewing the code, and I'm a bit confused
about the mpm_winnt pool cleanup code. I haven't spent a lot of time
reading the code in the past, so there's a good chance that I've just
missed something. As far as I can tell, ap_read_request() creates the
request pool, but nothing explicitly cleans it up. Instead, it looks like
mpm_recycle_completion_context() clears the ptrans pool the next time the
thread handles a request. While this seems funny to me, I don't see why
some of the memory would fail to be released.
I've tried to simplify my httpd.conf file to reduce the test case:
ServerRoot c:/varju/webct/webct/server
ThreadsPerChild 16
LoadModule cgi_module modules/mod_cgi.so
LoadModule alias_module modules/mod_alias.so
Listen 80
ScriptAlias /test.pl c:/varju/webct/webct/webct/generic/public/test.pl
Can anybody offer me suggestions of where to look next?
Thanks,
Alex.