Issue creating large # of nodes in a transaction. (SWIG perl bindings)

[Dan Mercer <dm...@8kb.net>]

Hi folks,

I have a tool that imports files into a subversion repository. Each 
import can consist of a large number of files and they should be 
imported in the same transaction. Unfortunately, my code is consistently 
crashing on any transaction that attempts to create > ~1400 new nodes 
(files or directories). Here are the particulars:

* Subversion version 1.3.0 (r17949)
* FSFS Repository
* Swig perl bindings (Perl 5.8.5)
* FreeBSD 4.11


And here is a very simple recipe to reproduce what I am seeing. Run on a
newly created fsfs repository as '$script /path/to/repos $count':

#!/usr/local/bin/perl -w

use strict;

use SVN::Core;
use SVN::Fs;
use SVN::Repos;

$|=1;

my $repos_path = shift;
my $count = shift;

my $svn = SVN::Repos::open($repos_path);
my $youngest = $svn->fs->youngest_rev();
my $rev_root = $svn->fs->revision_root($youngest);
my $txn = $svn->fs_begin_txn_for_commit($youngest, 'nobody', '');
my $txn_root = SVN::Fs::txn_root($txn);

my $nc = 0;
while ($nc < $count) {
    my $path = "/".$nc."_node";
    my $path_type = SVN::Fs::check_path($rev_root, $path);
    if ($path_type == $SVN::Node::none) {
#        SVN::Fs::make_dir($txn_root, $path);
        SVN::Fs::make_file($txn_root, $path);
        print "$nc,",
    }
    $nc++;
}
my $revnum = $svn->fs_commit_txn($txn);
print "New revision -> $revnum\n";

--

output:

$ svnadmin create --fs-type fsfs /tmp/test1
$ ./test.pl /tmp/test2 2048
0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,
...SNIP...
1406,1407,1408,1409,1410,1411,Abort trap (core dumped)
$
$ ./test.pl /tmp/test

I'm not sure what I'm doing wrong here, but any pointers would be 
helpful, even if you can just verify that this code behaves the same in 
your environment. It could be that my assumption that a single 
transaction should be able to handle many (certainly more than a few 
thousands) events is incorrect as well.

 

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org

Re: Issue creating large # of nodes in a transaction. (SWIG perl bindings)

[Dan Mercer <dm...@8kb.net>]

Another data point:

The following recipe using the swig python bindings (python 2.4.1) seems 
to work fine. I can use it to create 10,000 nodes in a single transaction:

#!/usr/local/bin/python -u

import sys
sys.path.append('/home/y/lib/svn-python')
from svn import core, fs, repos

if len(sys.argv) < 3:
    print "Usage: %s repos_path count" % (sys.argv[0])
    sys.exit(1)

svn = repos.open(sys.argv[1])
svn_fs  = repos.fs(svn)
youngest = fs.youngest_rev(svn_fs)
rev_root = fs.revision_root(svn_fs, youngest)
txn = repos.fs_begin_txn_for_commit(svn, youngest, 'nobody', '')
txn_root = fs.txn_root(txn)

nc = 0
while (nc < int(sys.argv[2])):
    path = "/%d_node" % nc
    path_type = fs.check_path(rev_root, path)
    if (path_type is core.svn_node_none):
        fs.make_dir(txn_root, path)
        print "%d," %nc,
    nc += 1

revnum = repos.fs_commit_txn(svn, txn)
print "New revision -> %d" % revnum

To me, this suggests some problem in the perl swig bindings. Any agreement?

Best,

Dan M

Dan Mercer wrote:
> Hi folks,
>
> I have a tool that imports files into a subversion repository. Each 
> import can consist of a large number of files and they should be 
> imported in the same transaction. Unfortunately, my code is 
> consistently crashing on any transaction that attempts to create > 
> ~1400 new nodes (files or directories). Here are the particulars:
>
> * Subversion version 1.3.0 (r17949)
> * FSFS Repository
> * Swig perl bindings (Perl 5.8.5)
> * FreeBSD 4.11
>
>
> And here is a very simple recipe to reproduce what I am seeing. Run on a
> newly created fsfs repository as '$script /path/to/repos $count':
>
> #!/usr/local/bin/perl -w
>
> use strict;
>
> use SVN::Core;
> use SVN::Fs;
> use SVN::Repos;
>
> $|=1;
>
> my $repos_path = shift;
> my $count = shift;
>
> my $svn = SVN::Repos::open($repos_path);
> my $youngest = $svn->fs->youngest_rev();
> my $rev_root = $svn->fs->revision_root($youngest);
> my $txn = $svn->fs_begin_txn_for_commit($youngest, 'nobody', '');
> my $txn_root = SVN::Fs::txn_root($txn);
>
> my $nc = 0;
> while ($nc < $count) {
>    my $path = "/".$nc."_node";
>    my $path_type = SVN::Fs::check_path($rev_root, $path);
>    if ($path_type == $SVN::Node::none) {
> #        SVN::Fs::make_dir($txn_root, $path);
>        SVN::Fs::make_file($txn_root, $path);
>        print "$nc,",
>    }
>    $nc++;
> }
> my $revnum = $svn->fs_commit_txn($txn);
> print "New revision -> $revnum\n";
>
> -- 
>
> output:
>
> $ svnadmin create --fs-type fsfs /tmp/test1
> $ ./test.pl /tmp/test2 2048
> 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,
> ...SNIP...
> 1406,1407,1408,1409,1410,1411,Abort trap (core dumped)
> $
> $ ./test.pl /tmp/test
>
> I'm not sure what I'm doing wrong here, but any pointers would be 
> helpful, even if you can just verify that this code behaves the same 
> in your environment. It could be that my assumption that a single 
> transaction should be able to handle many (certainly more than a few 
> thousands) events is incorrect as well.
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
> For additional commands, e-mail: users-help@subversion.tigris.org


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org

Re: Issue creating large # of nodes in a transaction. (SWIG perl bindings)

[Dan Mercer <dm...@8kb.net>]

Garrett Rooney wrote:
> On 3/10/06, Dan Mercer <dm...@8kb.net> wrote:
>> I'm not sure what I'm doing wrong here, but any pointers would be
>> helpful, even if you can just verify that this code behaves the same in
>> your environment. 
>
> You're probably running out of memory.  For a nontrivial use of the
> bindings you're going to have to do some pool management, see the docs
> in SVN::Core.pm for more details.
Thanks for the reminder. I've gotten spoiled not having to worry about 
pool allocation under the python bindings thanks to the improvements 
made by David James and others.

The following code works a treat:

#!/usr/local/bin/perl -w

use strict;

use SVN::Core;
use SVN::Fs;
use SVN::Repos;

$|=1;

my $repos_path = shift;
my $count = shift;

my $pool = SVN::Pool->new_default;

my $svn = SVN::Repos::open($repos_path);
my $youngest = $svn->fs->youngest_rev();
my $rev_root = $svn->fs->revision_root($youngest);
my $txn = $svn->fs_begin_txn_for_commit($youngest, 'nobody', '');
my $txn_root = SVN::Fs::txn_root($txn);

my $nc = 0;
while ($nc < $count) {
    my $pool = SVN::Pool->new_default_sub;
    my $path = "/".$nc."_node";
    my $path_type = SVN::Fs::check_path($rev_root, $path, $pool);
    if ($path_type == $SVN::Node::none) {
#        SVN::Fs::make_dir($txn_root, $path);
        SVN::Fs::make_file($txn_root, $path, $pool);
        print "$nc,",
    }
    $nc++;
}
my $revnum = $svn->fs_commit_txn($txn);
print "New revision -> $revnum\n";

I'll make the change in my tool and we'll be on our way.

Thanks,

Dan

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org

Re: Issue creating large # of nodes in a transaction. (SWIG perl bindings)

[Garrett Rooney <ro...@electricjellyfish.net>]

On 3/10/06, Dan Mercer <dm...@8kb.net> wrote:

> I'm not sure what I'm doing wrong here, but any pointers would be
> helpful, even if you can just verify that this code behaves the same in
> your environment. It could be that my assumption that a single
> transaction should be able to handle many (certainly more than a few
> thousands) events is incorrect as well.

You're probably running out of memory.  For a nontrivial use of the
bindings you're going to have to do some pool management, see the docs
in SVN::Core.pm for more details.

-garrett

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org