You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Philipp Marek <ph...@marek.priv.at> on 2007/10/02 12:29:10 UTC
Memory leak in svn_ra_get_file() with svn+ssh://?
Hello everybody,
I think there's a memory leak in svn_ra_get_file() or the svn+ssh RA layer.
FSVS (http://fsvs.tigris.org) uses svn_ra_get_file() eg. for reverts -
to fetch a clean copy of a file from the repository.
While it behaves for file://-URLs (memory usage constant), it grows
for svn+ssh://-accesses; when fetching a 100MB random-bytes file,
it takes 294MB more memory to finish that operation.
I've got a (partly useful) callstack:
#0 0x00002afb38d69cd0 in malloc () from /lib/libc.so.6
#1 0x00002afb391834dc in apr_palloc () from /usr/lib/libapr-1.so.0
#2 0x00002afb38387ea6 in svn_stringbuf_ensure () from /usr/lib/libsvn_subr-1.so.1
#3 0x00002afb38387f32 in svn_stringbuf_appendbytes () from /usr/lib/libsvn_subr-1.so.1
#4 0x00002afb3a21258f in ?? () from /usr/lib/libsvn_ra_svn-1.so.1
#5 0x00002afb3a21264d in svn_ra_svn_read_item () from /usr/lib/libsvn_ra_svn-1.so.1
#6 0x00002afb3a20bacc in ?? () from /usr/lib/libsvn_ra_svn-1.so.1
#7 0x0000000000413ac7 in rev__get_file (sts=0x63baf0, revision=10, fetched=0x7fff7295ec08, only_tmp=0x0, pool=<value optimized out>) at revert.c:275
... and so on up to main.
The last line (revert.c:275) is where svn_ra_get_file() gets called.
I have to admit that the stream is not a simple svn_stream_from_aprfile(),
but has a layer for blockwise MD5-hashing plugged in the middle; but
that should be the same for file:/// and svn+ssh:///.
Re: Memory leak in svn_ra_get_file() with svn+ssh://?
Posted by Philipp Marek <ph...@marek.priv.at>.
On Saturday 06 October 2007 David Glasser wrote:
> On 10/6/07, Philipp Marek <ph...@marek.priv.at> wrote:
> > Could you *please* *please* *please* change the read size to something
> > higher - how about 128k instead of the 4k we currently have?
> >
> > Similar for the ra_file layer ... there's 16k, but I think that is much
> > too low for current machines; we loose too much on context switches.
>
> This seems like the sort of thing that is most convincing with a
> benchmark. Care to write up a script we can run against servers of
> various block sizes and see which is most efficient?
How about using the recipe I had for the memory leak, possibly with a bigger
file? Using null bytes instead of random data would make no difference here,
too.
Example:
svnadmin create test
svn co file:///.../test wc
dd if=/dev/null of=wc/1 bs=1024k seek=640
svn add wc/1
svn ci wc -m1
And then do
time svn cat file:///...test/1 > /dev/null
with different block sizes.
(Or better, test with svn+ssh:// too).
Regards,
Phil
--
Versioning your /etc, /home or even your whole installation?
Try fsvs (fsvs.tigris.org)!
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Memory leak in svn_ra_get_file() with svn+ssh://?
Posted by David Glasser <gl...@davidglasser.net>.
On 10/6/07, Philipp Marek <ph...@marek.priv.at> wrote:
> Could you *please* *please* *please* change the read size to something
> higher - how about 128k instead of the 4k we currently have?
>
> Similar for the ra_file layer ... there's 16k, but I think that is much too
> low for current machines; we loose too much on context switches.
This seems like the sort of thing that is most convincing with a
benchmark. Care to write up a script we can run against servers of
various block sizes and see which is most efficient?
--dave
--
David Glasser | glasser@davidglasser.net | http://www.davidglasser.net/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Memory leak in svn_ra_get_file() with svn+ssh://?
Posted by Philipp Marek <ph...@marek.priv.at>.
Hello Dave!
On Saturday 06 October 2007 David Glasser wrote:
> On 10/5/07, David Glasser <gl...@davidglasser.net> wrote:
> > On 10/2/07, Philipp Marek <ph...@marek.priv.at> wrote:
> > > Hello everybody,
> > >
> > > I think there's a memory leak in svn_ra_get_file() or the svn+ssh RA
> > > layer. FSVS (http://fsvs.tigris.org) uses svn_ra_get_file() eg. for
> > > reverts - to fetch a clean copy of a file from the repository.
> >
> > Thanks for the bug report! It does in fact look to me like there is a
> > memory leak in libsvn_ra_svn/client.c(ra_svn_get_file). I'll commit
> > this as soon as tests finish passing (hopefully this will be before I
> > go home today).
>
> Committed in r26964, nominated for backport in r26965. Will be
> reverted when I find out my currently-running test suite fails :)
Could you *please* *please* *please* change the read size to something
higher - how about 128k instead of the 4k we currently have?
Similar for the ra_file layer ... there's 16k, but I think that is much too
low for current machines; we loose too much on context switches.
Regards,
Phil
--
Versioning your /etc, /home or even your whole installation?
Try fsvs (fsvs.tigris.org)!
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Memory leak in svn_ra_get_file() with svn+ssh://?
Posted by David Glasser <gl...@davidglasser.net>.
On 10/5/07, David Glasser <gl...@davidglasser.net> wrote:
> On 10/2/07, Philipp Marek <ph...@marek.priv.at> wrote:
> > Hello everybody,
> >
> > I think there's a memory leak in svn_ra_get_file() or the svn+ssh RA layer.
> > FSVS (http://fsvs.tigris.org) uses svn_ra_get_file() eg. for reverts -
> > to fetch a clean copy of a file from the repository.
>
> Thanks for the bug report! It does in fact look to me like there is a
> memory leak in libsvn_ra_svn/client.c(ra_svn_get_file). I'll commit
> this as soon as tests finish passing (hopefully this will be before I
> go home today).
Committed in r26964, nominated for backport in r26965. Will be
reverted when I find out my currently-running test suite fails :)
--dave
--
David Glasser | glasser@davidglasser.net | http://www.davidglasser.net/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Memory leak in svn_ra_get_file() with svn+ssh://?
Posted by David Glasser <gl...@davidglasser.net>.
On 10/2/07, Philipp Marek <ph...@marek.priv.at> wrote:
> Hello everybody,
>
> I think there's a memory leak in svn_ra_get_file() or the svn+ssh RA layer.
> FSVS (http://fsvs.tigris.org) uses svn_ra_get_file() eg. for reverts -
> to fetch a clean copy of a file from the repository.
Thanks for the bug report! It does in fact look to me like there is a
memory leak in libsvn_ra_svn/client.c(ra_svn_get_file). I'll commit
this as soon as tests finish passing (hopefully this will be before I
go home today).
[[[
Fix unbounded memory usage in ra_svn implementation of
svn_ra_get_file. ('svn cat'
Found by: pmarek
* subversion/libsvn_ra_svn/client.c
(ra_svn_get_file): Introduce an iterpool into the loop that reads
file contents from the server 4K at a time.
]]]
Index: subversion/libsvn_ra_svn/client.c
===================================================================
--- subversion/libsvn_ra_svn/client.c (revision 26963)
+++ subversion/libsvn_ra_svn/client.c (working copy)
@@ -877,6 +877,7 @@ static svn_error_t *ra_svn_get_file(svn_
unsigned char digest[APR_MD5_DIGESTSIZE];
const char *expected_checksum, *hex_digest;
apr_md5_ctx_t md5_context;
+ apr_pool_t *iterpool;
SVN_ERR(svn_ra_svn_write_cmd(conn, pool, "get-file", "c(?r)bb", path,
rev, (props != NULL), (stream != NULL)));
@@ -898,9 +899,11 @@ static svn_error_t *ra_svn_get_file(svn_
apr_md5_init(&md5_context);
/* Read the file's contents. */
+ iterpool = svn_pool_create(pool);
while (1)
{
- SVN_ERR(svn_ra_svn_read_item(conn, pool, &item));
+ svn_pool_clear(iterpool);
+ SVN_ERR(svn_ra_svn_read_item(conn, iterpool, &item));
if (item->kind != SVN_RA_SVN_STRING)
return svn_error_create(SVN_ERR_RA_SVN_MALFORMED_DATA, NULL,
_("Non-string as part of file contents"));
@@ -914,6 +917,8 @@ static svn_error_t *ra_svn_get_file(svn_
SVN_ERR(svn_stream_write(stream, item->u.string->data,
&item->u.string->len));
}
+ svn_pool_destroy(iterpool);
+
SVN_ERR(svn_ra_svn_read_cmd_response(conn, pool, ""));
if (expected_checksum)
--
David Glasser | glasser@davidglasser.net | http://www.davidglasser.net/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org