You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by Bill Stoddard <bi...@wstoddard.com> on 2002/09/04 23:48:13 UTC
Source of memory leak in mod_proxy?
I just got a bug report that that the rewrite of ap_get_client_block() has
introduced a memory leak. Problem is observed when putting a large file to
the server via webdav. Replacing the 2.0.40 implementation with the 2.0.39
implementation (on a 2.0.40 code base) eliminates the problem. I do not have
time to investigate this now. I can take a look tomorrow or perhaps this
evening if no one wants to pick it up.
http://cvs.apache.org/viewcvs.cgi/httpd-2.0/modules/http/http_protocol.c.dif
f?r1=1.436&r2=1.437
Bill
Re: Source of memory leak in mod_proxy?
Posted by Brian Pane <br...@apache.org>.
Peter Van Biesen wrote:
>If you are talking about the memory leak when downloading large files
>through a proxy chain, this problem was first observed in version
>2.0.39, so it probably isn't the same problem.
>
2.0.39 definitely would have had problems with large requests
due to the way its content-length filter worked. Different part
of the code, but the same basic problem: setting aside arbitrarily
large amounts of data.
Brian
Re: Source of memory leak in mod_proxy?
Posted by Peter Van Biesen <pe...@vlafo.be>.
If you are talking about the memory leak when downloading large files
through a proxy chain, this problem was first observed in version
2.0.39, so it probably isn't the same problem.
Peter.
Bill Stoddard wrote:
>
> I just got a bug report that that the rewrite of ap_get_client_block() has
> introduced a memory leak. Problem is observed when putting a large file to
> the server via webdav. Replacing the 2.0.40 implementation with the 2.0.39
> implementation (on a 2.0.40 code base) eliminates the problem. I do not have
> time to investigate this now. I can take a look tomorrow or perhaps this
> evening if no one wants to pick it up.
>
> http://cvs.apache.org/viewcvs.cgi/httpd-2.0/modules/http/http_protocol.c.dif
> f?r1=1.436&r2=1.437
>
> Bill
Re: Source of memory leak in mod_proxy?
Posted by Cliff Woolley <jw...@virginia.edu>.
On Wed, 4 Sep 2002, Justin Erenkrantz wrote:
> Will the buckets/brigade be cleaned up when the request_rec's
> pool is destroyed or are do they live as long as the connection
> pool?
Buckets live forever until you destroy them.
Brigades live as long as the pool they're in (normally r->pool) and
automatically delete any buckets still in them when that pool cleans up
The bucket allocator will allocate as much memory as is needed for the
maximum amount of simultaneously-alive buckets *ever*. c->bucket_alloc
is a pointer to an allocator that may live as long as the whole thread.
In other words, the easiest ways to leak memory with buckets are to take a
bunch of buckets out of their brigade and then forget about them, or leave
them in the brigade and then forget about the brigade, in which case
they'll at least get cleaned up when r->pool goes away, but that still
might result in a lot of buckets simultaneously allocated before the
request is done. Or you could have a filter that buffers way more stuff
than it should, but that's usually a more obvious bug.
> I'm purposely *not* looking at the code as I'd like to know the
> intention rather than what we actually do. =) -- justin
Me too. :)
Does the above help?
--Cliff
Re: Source of memory leak in mod_proxy?
Posted by Justin Erenkrantz <je...@apache.org>.
On Wed, Sep 04, 2002 at 10:11:13PM -0400, Cliff Woolley wrote:
> on some bucket that you're actually finished with. This could be a
> symptom of discarding a brigade that still has buckets in it (since
> apr_brigade_destroy() will destroy all the buckets that are in the brigade
> at the time of the brigade's destruction), which is, I believe, what you
> guys have said is happening in this case.
Will the buckets/brigade be cleaned up when the request_rec's
pool is destroyed or are do they live as long as the connection
pool?
It has ties to both r->pool and r->connection->bucket_alloc. I'm
purposely *not* looking at the code as I'd like to know the
intention rather than what we actually do. =) -- justin
Re: Source of memory leak in mod_proxy?
Posted by Brian Pane <br...@apache.org>.
Cliff Woolley wrote:
>Bottom line: the size of c->bucket_alloc will be based on the max number
>of buckets and bucket buffers you've had allocated at one time. The
>answer is almost certainly that apr_bucket_destroy() is not being called
>on some bucket that you're actually finished with. This could be a
>symptom of discarding a brigade that still has buckets in it (since
>apr_brigade_destroy() will destroy all the buckets that are in the brigade
>at the time of the brigade's destruction), which is, I believe, what you
>guys have said is happening in this case.
>
>
>
Yep, we really need to clean up that temp brigade in ap_get_client_block().
I'm testing a fix now; I'll commit it soon if the fix works.
Brian
Re: Source of memory leak in mod_proxy?
Posted by Cliff Woolley <jw...@virginia.edu>.
On Wed, 4 Sep 2002, Greg Stein wrote:
> bb = apr_brigade_create(r->pool, r->connection->bucket_alloc); Does
> that allocator somehow use connection memory? Maybe we have some kind of
> leak between the request-based brigade and the connection-based
> allocator? [ I don't know enough about the bucket allocator stuff ]
The purpose is supposed to be that, over the course of the connections for
which the allocator is used, you reach a steady state where n-bytes are
the most needed at one time. Unlike the connection pool, the
c->bucket_alloc reuses memory as buckets get freed.
Bottom line: the size of c->bucket_alloc will be based on the max number
of buckets and bucket buffers you've had allocated at one time. The
answer is almost certainly that apr_bucket_destroy() is not being called
on some bucket that you're actually finished with. This could be a
symptom of discarding a brigade that still has buckets in it (since
apr_brigade_destroy() will destroy all the buckets that are in the brigade
at the time of the brigade's destruction), which is, I believe, what you
guys have said is happening in this case.
--Cliff
Re: Source of memory leak in mod_proxy?
Posted by Cliff Woolley <jw...@virginia.edu>.
On Wed, 4 Sep 2002, Greg Stein wrote:
> bb = apr_brigade_create(r->pool, r->connection->bucket_alloc); Does
> that allocator somehow use connection memory? Maybe we have some kind of
> leak between the request-based brigade and the connection-based
> allocator? [ I don't know enough about the bucket allocator stuff ]
The purpose is supposed to be that, over the course of the connections for
which the allocator is used, you reach a steady state where n-bytes are
the most needed at one time. Unlike the connection pool, the
c->bucket_alloc reuses memory as buckets get freed.
Bottom line: the size of c->bucket_alloc will be based on the max number
of buckets and bucket buffers you've had allocated at one time. The
answer is almost certainly that apr_bucket_destroy() is not being called
on some bucket that you're actually finished with. This could be a
symptom of discarding a brigade that still has buckets in it (since
apr_brigade_destroy() will destroy all the buckets that are in the brigade
at the time of the brigade's destruction), which is, I believe, what you
guys have said is happening in this case.
--Cliff
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Source of memory leak in mod_proxy?
Posted by Greg Stein <gs...@lyra.org>.
On Wed, Sep 04, 2002 at 05:24:07PM -0700, Justin Erenkrantz wrote:
> On Wed, Sep 04, 2002 at 05:17:29PM -0700, Greg Stein wrote:
> >
> > Yup. On the error exits, too.
>
> Hmm. On the error exits, shouldn't the request pool get cleaned up
> right after that? But, for the successful cases, it makes sense to
> cleanup.
Are you sure about that? Will ever caller to ap_get_client_block() exit and
clean up the request pool? :-)
> > You know... I bet this will also solve the 'svn import' problem that we've
> > been seeing lately. In that case, the server memory just skyrockets,
> > proportional to the amount of data imported. This would do it.
>
> Hmm. Why wouldn't they be cleaned up when the request pool
> is cleaned up? I wonder if its something else there. As,
> svn import is multiple-request based rather than one large
> commit, right? So, r->pool would be cleaned up on each request
> and the brigade freed.
>
> Or, maybe I'm missing something... -- justin
Oh... right. Hrm. But in that case, then SVN shouldn't be accumulating
memory either. "multiple requests"
Hey... look at this line:
bb = apr_brigade_create(r->pool, r->connection->bucket_alloc);
Does that allocator somehow use connection memory? Maybe we have some kind
of leak between the request-based brigade and the connection-based
allocator?
[ I don't know enough about the bucket allocator stuff ]
Alternatively, we might also be seeing some kind of fragmentation in the
allocator. Note that we might not be seeing the problem in other scenarios
simply because we're doing PUTs here (and most testing doesn't), and that
we're dealing with heap memory as opposed to FILE or PIPE buckets.
Cheers,
-g
--
Greg Stein, http://www.lyra.org/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Source of memory leak in mod_proxy?
Posted by Greg Stein <gs...@lyra.org>.
On Wed, Sep 04, 2002 at 05:24:07PM -0700, Justin Erenkrantz wrote:
> On Wed, Sep 04, 2002 at 05:17:29PM -0700, Greg Stein wrote:
> >
> > Yup. On the error exits, too.
>
> Hmm. On the error exits, shouldn't the request pool get cleaned up
> right after that? But, for the successful cases, it makes sense to
> cleanup.
Are you sure about that? Will ever caller to ap_get_client_block() exit and
clean up the request pool? :-)
> > You know... I bet this will also solve the 'svn import' problem that we've
> > been seeing lately. In that case, the server memory just skyrockets,
> > proportional to the amount of data imported. This would do it.
>
> Hmm. Why wouldn't they be cleaned up when the request pool
> is cleaned up? I wonder if its something else there. As,
> svn import is multiple-request based rather than one large
> commit, right? So, r->pool would be cleaned up on each request
> and the brigade freed.
>
> Or, maybe I'm missing something... -- justin
Oh... right. Hrm. But in that case, then SVN shouldn't be accumulating
memory either. "multiple requests"
Hey... look at this line:
bb = apr_brigade_create(r->pool, r->connection->bucket_alloc);
Does that allocator somehow use connection memory? Maybe we have some kind
of leak between the request-based brigade and the connection-based
allocator?
[ I don't know enough about the bucket allocator stuff ]
Alternatively, we might also be seeing some kind of fragmentation in the
allocator. Note that we might not be seeing the problem in other scenarios
simply because we're doing PUTs here (and most testing doesn't), and that
we're dealing with heap memory as opposed to FILE or PIPE buckets.
Cheers,
-g
--
Greg Stein, http://www.lyra.org/
Re: Source of memory leak in mod_proxy?
Posted by Justin Erenkrantz <je...@apache.org>.
On Wed, Sep 04, 2002 at 05:17:29PM -0700, Greg Stein wrote:
>
> Yup. On the error exits, too.
Hmm. On the error exits, shouldn't the request pool get cleaned up
right after that? But, for the successful cases, it makes sense to
cleanup.
> You know... I bet this will also solve the 'svn import' problem that we've
> been seeing lately. In that case, the server memory just skyrockets,
> proportional to the amount of data imported. This would do it.
Hmm. Why wouldn't they be cleaned up when the request pool
is cleaned up? I wonder if its something else there. As,
svn import is multiple-request based rather than one large
commit, right? So, r->pool would be cleaned up on each request
and the brigade freed.
Or, maybe I'm missing something... -- justin
Re: Source of memory leak in mod_proxy?
Posted by Justin Erenkrantz <je...@apache.org>.
On Wed, Sep 04, 2002 at 05:17:29PM -0700, Greg Stein wrote:
>
> Yup. On the error exits, too.
Hmm. On the error exits, shouldn't the request pool get cleaned up
right after that? But, for the successful cases, it makes sense to
cleanup.
> You know... I bet this will also solve the 'svn import' problem that we've
> been seeing lately. In that case, the server memory just skyrockets,
> proportional to the amount of data imported. This would do it.
Hmm. Why wouldn't they be cleaned up when the request pool
is cleaned up? I wonder if its something else there. As,
svn import is multiple-request based rather than one large
commit, right? So, r->pool would be cleaned up on each request
and the brigade freed.
Or, maybe I'm missing something... -- justin
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Source of memory leak in mod_proxy?
Posted by Greg Stein <gs...@lyra.org>.
On Wed, Sep 04, 2002 at 02:53:53PM -0700, Justin Erenkrantz wrote:
> On Wed, Sep 04, 2002 at 05:48:13PM -0400, Bill Stoddard wrote:
> > I just got a bug report that that the rewrite of ap_get_client_block() has
> > introduced a memory leak. Problem is observed when putting a large file to
> > the server via webdav. Replacing the 2.0.40 implementation with the 2.0.39
> > implementation (on a 2.0.40 code base) eliminates the problem. I do not have
> > time to investigate this now. I can take a look tomorrow or perhaps this
> > evening if no one wants to pick it up.
[ Bill refers to revision 1.437 of http_protocol.c ]
> Ah, I bet it should destroy the brigade after flattening. -- justin
Yup. On the error exits, too.
You know... I bet this will also solve the 'svn import' problem that we've
been seeing lately. In that case, the server memory just skyrockets,
proportional to the amount of data imported. This would do it.
[ see http://subversion.tigris.org/issues/show_bug.cgi?id=860 ]
Cheers,
-g
--
Greg Stein, http://www.lyra.org/
Re: Source of memory leak in mod_proxy?
Posted by Greg Stein <gs...@lyra.org>.
On Wed, Sep 04, 2002 at 02:53:53PM -0700, Justin Erenkrantz wrote:
> On Wed, Sep 04, 2002 at 05:48:13PM -0400, Bill Stoddard wrote:
> > I just got a bug report that that the rewrite of ap_get_client_block() has
> > introduced a memory leak. Problem is observed when putting a large file to
> > the server via webdav. Replacing the 2.0.40 implementation with the 2.0.39
> > implementation (on a 2.0.40 code base) eliminates the problem. I do not have
> > time to investigate this now. I can take a look tomorrow or perhaps this
> > evening if no one wants to pick it up.
[ Bill refers to revision 1.437 of http_protocol.c ]
> Ah, I bet it should destroy the brigade after flattening. -- justin
Yup. On the error exits, too.
You know... I bet this will also solve the 'svn import' problem that we've
been seeing lately. In that case, the server memory just skyrockets,
proportional to the amount of data imported. This would do it.
[ see http://subversion.tigris.org/issues/show_bug.cgi?id=860 ]
Cheers,
-g
--
Greg Stein, http://www.lyra.org/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Source of memory leak in mod_proxy?
Posted by Justin Erenkrantz <je...@apache.org>.
On Wed, Sep 04, 2002 at 05:48:13PM -0400, Bill Stoddard wrote:
> I just got a bug report that that the rewrite of ap_get_client_block() has
> introduced a memory leak. Problem is observed when putting a large file to
> the server via webdav. Replacing the 2.0.40 implementation with the 2.0.39
> implementation (on a 2.0.40 code base) eliminates the problem. I do not have
> time to investigate this now. I can take a look tomorrow or perhaps this
> evening if no one wants to pick it up.
Ah, I bet it should destroy the brigade after flattening. -- justin
Re: Source of memory leak in mod_proxy?
Posted by Peter Van Biesen <pe...@vlafo.be>.
If you are talking about the memory leak when downloading large files
through a proxy chain, this problem was first observed in version
2.0.39, so it probably isn't the same problem.
Peter.
Bill Stoddard wrote:
>
> I just got a bug report that that the rewrite of ap_get_client_block() has
> introduced a memory leak. Problem is observed when putting a large file to
> the server via webdav. Replacing the 2.0.40 implementation with the 2.0.39
> implementation (on a 2.0.40 code base) eliminates the problem. I do not have
> time to investigate this now. I can take a look tomorrow or perhaps this
> evening if no one wants to pick it up.
>
> http://cvs.apache.org/viewcvs.cgi/httpd-2.0/modules/http/http_protocol.c.dif
> f?r1=1.436&r2=1.437
>
> Bill