Avoiding leaks when using SVN::Fs, etc.

[Dan Mercer <dm...@8kb.net>]

Hi folks,

I have some tools that make use of the perl bindings to Subversion 1.3. 
As we are dealing with moderate amounts of data in long running 
processes, we are managing memory using the SVN::Pool interface but it 
appears that we are leaking memory. The leak I am focusing on right now 
is related to how I am calling SVN::Fs::file_contents, (though I have 
found other leaks in at least my usage of apply_textdelta and send_string).

I am including a simple test script which triggers the same behavior 
exhibited in the actual tool. The script simply iterates through $count 
creations of a svn repos, testing for and getting a stream for the file 
at $path. If $leak is set, the stream object is read and the contents 
assigned to a variable, causing some number of bytes to be leaked -- 
otherwise if no read happens, no memory leaks. Here is examples of the 
summary output:

# 10 iterations without reading from the file stream:

$ ./svn_find_leak.pl /tmp/test6 /1/roles 10
Starting memory usage: 5628
Ending memory usage:   5884
growth in 10 iterations: 256 bytes

# 1000 iterations without reading from the file stream:

$ ./svn_find_leak.pl /tmp/test6 /1/roles 1000
Starting memory usage: 5628
Ending memory usage:   5884
growth in 100 iterations: 256 bytes

# 10 iterations reading from the file stream:

$ ./svn_find_leak.pl /tmp/test6 /1/roles 10 leak
Starting memory usage: 5628
Ending memory usage:   5948
growth in 10 iterations: 320 bytes

# 1000 iterations reading from the file stream:

$ ./svn_find_leak.pl /tmp/test6 /1/roles 1000 leak
Starting memory usage: 5628
Ending memory usage:   11904
growth in 1000 iterations: 6276 bytes

I think I am missing something in how I'm handling reading from the 
stream. In the SVN::Core manpage there is this warning for SVN::Stream:

  Note that some functions take a stream to read or write, while it does
  not close it but still hold the reference to the handle. In this case
  the handle won't be destroyed properly. You should always use correct
  default pool before calling such functions.

Unfortunately, I don't understand this statement. :-) It seems to 
indicate that one really ought to be making use of implicit 'default' 
pools rather than keeping track of them yourself. I *do* know that our 
early experimentation with using the 'default' pool behavior available 
from SVN::Pool did not seem to work as expected. Pools marked as default 
cleared through $pool->clear() or undefined were not removed from the 
poolstack, which would simply continue to grow in depth as the process run.
I feel more comfortable explicitly allocating pools and passing them to 
every SVN:: function, but if there are examples of better patterns, I am 
willing to give them a try. Right now the best tips have come from 
Garrett Rooney's article on the SVN client API and Greg Steins notes on 
best practices linked from apr.apache.org.

Here is the test script. It will only work on Unix like systems and if 
your ps works remotely like ours does. (seems to work (and leak) on both 
RHEL 4 and FreeBSD 4.11). Also, we are using Perl 5.8.5 and 5.6.1.

#!/usr/local/bin/perl -w
use strict;
use lib qw(/home/y/lib/perl5/site_perl);
use SVN::Core;
use SVN::Repos;
use SVN::Fs;

my $repos = shift;   # arg 1 - path to a subversion repository
my $path  = shift;   # arg 2 - path in repository to some file (must exist)
my $count = shift;   # arg 3 - # of iterations to run for
my $leak = shift;    # arg 4 - set to something to leak

my $start_mem = get_mem();
my $prev_mem = $start_mem;

for (0..$count) {

    my $p = SVN::Pool->new(undef);
    my $svn = SVN::Repos::open($repos, $p);
    my $fs  = $svn->fs();
    my $youngest = $fs->youngest_rev($p);
    my $root = $fs->revision_root($youngest, $p);
    my $type = SVN::Fs::check_path($root, $path, $p);
    if ($type == $SVN::Core::node_none) {
        die "The file $path does not exist in $repos @ r$youngest!";
    } elsif ($type == $SVN::Core::node_dir) {
        die "$path is a directory in $repos @ r$youngest!";
    }
    # allocate a subpool for operations on this file. this is so that
    # the test looks like our actual use-case, but if you do away with
    # this and just allocate from $p, the leak will still occur.
    my $sp = SVN::Pool->new($p);
    my $stream = SVN::Fs::file_contents($root, $path, $sp);
    # we only leak if we try to read from $stream
    if (defined $leak) {
        my $contents = <$stream>;
    }
    # cleanup
    $stream = undef;
    $sp = undef;
    $svn = undef;
    $p = undef;

    my $mem = get_mem();
    print "memory usage $mem\n";
    print "memory growth ",$mem - $prev_mem, " bytes \n";
    $prev_mem = $mem;
    print "=" x 75, "\n";
}

my $end_mem = get_mem();
my $increase = ($end_mem - $start_mem);
print "Starting memory usage: $start_mem\n";
print "Ending memory usage:   $end_mem\n";
print "growth in $count iterations: $increase bytes\n";

sub get_mem {
    my $mem = `ps -o rss -p $$ | tail -1`;
    chomp($mem);
    return $mem;
}

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org

Re: Avoiding leaks when using SVN::Fs, etc.

[Dan Mercer <dm...@8kb.net>]

Dan Mercer wrote:
> Hi folks,
>
> I have some tools that make use of the perl bindings to Subversion 
> 1.3. As we are dealing with moderate amounts of data in long running 
> processes, we are managing memory using the SVN::Pool interface but it 
> appears that we are leaking memory. The leak I am focusing on right 
> now is related to how I am calling SVN::Fs::file_contents, (though I 
> have found other leaks in at least my usage of apply_textdelta and 
> send_string).

The leak I described appears be in the getline method of SVN:Stream (in 
Core.pm):

sub getline
{
    my $self = shift;
    *$self->{pool} ||= SVN::Core::pool_create (undef);
    my ($buf, $eof) = *$self->{svn_stream}->readline ($/, *$self->{pool});
    return undef if $eof && !length($buf);
    return $eof ? $buf : $buf.$/;
}

It appears that SVN::Stream, $self->{pool} will never exist and so is 
always allocated for the read. This memory is not cleaned up, even if a 
default pool is created prior to the read. Fortunately, file contents 
can be read using the 'read' method in a way that doesn't cause a memory 
leak:

my $stream = SVN::Fs::file_contents($root, $path, $pool);
my $size = SVN::Fs::file_length($root, $path, $pool);
my $data = 
'';                                                                 # 
must be a scalar and not undef!
$stream->read($data, $size);                                         # 
$data contains file contents

I am including a patch to my earlier script that demonstrates that 
<$stream> still leaks, even if a subpool has been explicitly created. It 
also demonstrates a non-leaky read of file contents using the read method.

--- svn_find_leak.pl    Thu Apr 13 10:50:24 2006
+++ svn_find_leak2.pl   Thu Apr 13 12:43:34 2006
@@ -30,13 +30,20 @@
    # the test looks like our actual use-case, but if you do away with
    # this and just allocate from $p, the leak will still occur.
    my $sp = SVN::Pool->new($p);
-   my $stream = SVN::Fs::file_contents($root, $path, $sp);
    # we only leak if we try to read from $stream
    if (defined $leak) {
+       $sp->default();
+       my $stream = SVN::Fs::file_contents($root, $path, $sp);
        my $contents = <$stream>;
+       $stream = undef;
+   } else {
+       my $contents = '';
+       my $stream = SVN::Fs::file_contents($root, $path, $sp);
+       my $size = SVN::Fs::file_length($root, $path, $sp);
+       $stream->read($contents, $size);
+       $stream = undef;
    }
    # cleanup
-   $stream = undef;
    $sp = undef;

Hopefully this will be helpful to those making use of the perl bindings 
to SVN. I would submit a patch for getlines(), but I'm not really sure 
what the appropriate fix should be... maybe this will help someone with 
greater perl skills than myself devise a real fix.

Dan M.

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org