You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Philipp Marek <ph...@marek.priv.at> on 2007/10/02 12:29:10 UTC

Memory leak in svn_ra_get_file() with svn+ssh://?

Hello everybody,

I think there's a memory leak in svn_ra_get_file() or the svn+ssh RA layer.
FSVS (http://fsvs.tigris.org) uses svn_ra_get_file() eg. for reverts - 
to fetch a clean copy of a file from the repository.

While it behaves for file://-URLs (memory usage constant), it grows 
for svn+ssh://-accesses; when fetching a 100MB random-bytes file, 
it takes 294MB more memory to finish that operation.

I've got a (partly useful) callstack:

#0  0x00002afb38d69cd0 in malloc () from /lib/libc.so.6
#1  0x00002afb391834dc in apr_palloc () from /usr/lib/libapr-1.so.0
#2  0x00002afb38387ea6 in svn_stringbuf_ensure () from /usr/lib/libsvn_subr-1.so.1
#3  0x00002afb38387f32 in svn_stringbuf_appendbytes () from /usr/lib/libsvn_subr-1.so.1
#4  0x00002afb3a21258f in ?? () from /usr/lib/libsvn_ra_svn-1.so.1
#5  0x00002afb3a21264d in svn_ra_svn_read_item () from /usr/lib/libsvn_ra_svn-1.so.1
#6  0x00002afb3a20bacc in ?? () from /usr/lib/libsvn_ra_svn-1.so.1
#7  0x0000000000413ac7 in rev__get_file (sts=0x63baf0, revision=10, fetched=0x7fff7295ec08, only_tmp=0x0, pool=<value optimized out>) at revert.c:275
... and so on up to main.

The last line (revert.c:275) is where svn_ra_get_file() gets called.
I have to admit that the stream is not a simple svn_stream_from_aprfile(), 
but has a layer for blockwise MD5-hashing plugged in the middle; but 
that should be the same for file:/// and svn+ssh:///.

Re: Memory leak in svn_ra_get_file() with svn+ssh://?

Posted by Philipp Marek <ph...@marek.priv.at>.
On Saturday 06 October 2007 David Glasser wrote:
> On 10/6/07, Philipp Marek <ph...@marek.priv.at> wrote:
> > Could you *please* *please* *please* change the read size to something
> > higher - how about 128k instead of the 4k we currently have?
> >
> > Similar for the ra_file layer ... there's 16k, but I think that is much
> > too low for current machines; we loose too much on context switches.
>
> This seems like the sort of thing that is most convincing with a
> benchmark.  Care to write up a script we can run against servers of
> various block sizes and see which is most efficient?
How about using the recipe I had for the memory leak, possibly with a bigger 
file? Using null bytes instead of random data would make no difference here, 
too.

Example:
	svnadmin create test
	svn co file:///.../test wc
	dd if=/dev/null of=wc/1 bs=1024k seek=640 
	svn add wc/1
	svn ci wc -m1

And then do 
	time svn cat file:///...test/1 > /dev/null
with different block sizes.


(Or better, test with svn+ssh:// too).


Regards,

Phil


-- 
Versioning your /etc, /home or even your whole installation?
             Try fsvs (fsvs.tigris.org)!

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: Memory leak in svn_ra_get_file() with svn+ssh://?

Posted by David Glasser <gl...@davidglasser.net>.
On 10/6/07, Philipp Marek <ph...@marek.priv.at> wrote:
> Could you *please* *please* *please* change the read size to something
> higher - how about 128k instead of the 4k we currently have?
>
> Similar for the ra_file layer ... there's 16k, but I think that is much too
> low for current machines; we loose too much on context switches.

This seems like the sort of thing that is most convincing with a
benchmark.  Care to write up a script we can run against servers of
various block sizes and see which is most efficient?

--dave

-- 
David Glasser | glasser@davidglasser.net | http://www.davidglasser.net/

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: Memory leak in svn_ra_get_file() with svn+ssh://?

Posted by Philipp Marek <ph...@marek.priv.at>.
Hello Dave!

On Saturday 06 October 2007 David Glasser wrote:
> On 10/5/07, David Glasser <gl...@davidglasser.net> wrote:
> > On 10/2/07, Philipp Marek <ph...@marek.priv.at> wrote:
> > > Hello everybody,
> > >
> > > I think there's a memory leak in svn_ra_get_file() or the svn+ssh RA
> > > layer. FSVS (http://fsvs.tigris.org) uses svn_ra_get_file() eg. for
> > > reverts - to fetch a clean copy of a file from the repository.
> >
> > Thanks for the bug report!  It does in fact look to me like there is a
> > memory leak in libsvn_ra_svn/client.c(ra_svn_get_file).  I'll commit
> > this as soon as tests finish passing (hopefully this will be before I
> > go home today).
>
> Committed in r26964, nominated for backport in r26965.  Will be
> reverted when I find out my currently-running test suite fails :)

Could you *please* *please* *please* change the read size to something 
higher - how about 128k instead of the 4k we currently have?

Similar for the ra_file layer ... there's 16k, but I think that is much too 
low for current machines; we loose too much on context switches.


Regards,

Phil


-- 
Versioning your /etc, /home or even your whole installation?
             Try fsvs (fsvs.tigris.org)!

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: Memory leak in svn_ra_get_file() with svn+ssh://?

Posted by David Glasser <gl...@davidglasser.net>.
On 10/5/07, David Glasser <gl...@davidglasser.net> wrote:
> On 10/2/07, Philipp Marek <ph...@marek.priv.at> wrote:
> > Hello everybody,
> >
> > I think there's a memory leak in svn_ra_get_file() or the svn+ssh RA layer.
> > FSVS (http://fsvs.tigris.org) uses svn_ra_get_file() eg. for reverts -
> > to fetch a clean copy of a file from the repository.
>
> Thanks for the bug report!  It does in fact look to me like there is a
> memory leak in libsvn_ra_svn/client.c(ra_svn_get_file).  I'll commit
> this as soon as tests finish passing (hopefully this will be before I
> go home today).

Committed in r26964, nominated for backport in r26965.  Will be
reverted when I find out my currently-running test suite fails :)

--dave

-- 
David Glasser | glasser@davidglasser.net | http://www.davidglasser.net/

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: Memory leak in svn_ra_get_file() with svn+ssh://?

Posted by David Glasser <gl...@davidglasser.net>.
On 10/2/07, Philipp Marek <ph...@marek.priv.at> wrote:
> Hello everybody,
>
> I think there's a memory leak in svn_ra_get_file() or the svn+ssh RA layer.
> FSVS (http://fsvs.tigris.org) uses svn_ra_get_file() eg. for reverts -
> to fetch a clean copy of a file from the repository.

Thanks for the bug report!  It does in fact look to me like there is a
memory leak in libsvn_ra_svn/client.c(ra_svn_get_file).  I'll commit
this as soon as tests finish passing (hopefully this will be before I
go home today).

[[[
Fix unbounded memory usage in ra_svn implementation of
svn_ra_get_file.  ('svn cat'

Found by: pmarek

* subversion/libsvn_ra_svn/client.c
  (ra_svn_get_file): Introduce an iterpool into the loop that reads
   file contents from the server 4K at a time.
]]]

Index: subversion/libsvn_ra_svn/client.c
===================================================================
--- subversion/libsvn_ra_svn/client.c	(revision 26963)
+++ subversion/libsvn_ra_svn/client.c	(working copy)
@@ -877,6 +877,7 @@ static svn_error_t *ra_svn_get_file(svn_
   unsigned char digest[APR_MD5_DIGESTSIZE];
   const char *expected_checksum, *hex_digest;
   apr_md5_ctx_t md5_context;
+  apr_pool_t *iterpool;

   SVN_ERR(svn_ra_svn_write_cmd(conn, pool, "get-file", "c(?r)bb", path,
                                rev, (props != NULL), (stream != NULL)));
@@ -898,9 +899,11 @@ static svn_error_t *ra_svn_get_file(svn_
     apr_md5_init(&md5_context);

   /* Read the file's contents. */
+  iterpool = svn_pool_create(pool);
   while (1)
     {
-      SVN_ERR(svn_ra_svn_read_item(conn, pool, &item));
+      svn_pool_clear(iterpool);
+      SVN_ERR(svn_ra_svn_read_item(conn, iterpool, &item));
       if (item->kind != SVN_RA_SVN_STRING)
         return svn_error_create(SVN_ERR_RA_SVN_MALFORMED_DATA, NULL,
                                 _("Non-string as part of file contents"));
@@ -914,6 +917,8 @@ static svn_error_t *ra_svn_get_file(svn_
       SVN_ERR(svn_stream_write(stream, item->u.string->data,
                                &item->u.string->len));
     }
+  svn_pool_destroy(iterpool);
+
   SVN_ERR(svn_ra_svn_read_cmd_response(conn, pool, ""));

   if (expected_checksum)

-- 
David Glasser | glasser@davidglasser.net | http://www.davidglasser.net/

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org