You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by "Martin J. Evans" <ma...@easysoft.com> on 2004/05/28 11:06:20 UTC
svn client fails with Berkeley DB error Cannot allocate memory - repository wedges
Hi,
Since no one has yet replied to my "tree view of repository including following
external references" posting I put together a small perl script using WEB::SVN
to do the job (see below). Running this program to a dav URL I see in the
Apache log:
./x.pl -u http://a.b.c/linux-x86
[Fri May 28 09:33:34 2004] [error] [client 192.168.0.78] Could not fetch resourc
e information. [500, #0]
[Fri May 28 09:33:34 2004] [error] [client 192.168.0.78] Could not open the requ
ested SVN filesystem [500, #160029]
[Fri May 28 09:33:34 2004] [error] [client 192.168.0.78] (17)File exists: Could
not open the requested SVN filesystem [500, #160029]
[Fri May 28 09:34:04 2004] [error] [client 192.168.0.78] (20014)Error string not
specified yet: Berkeley DB error while opening 'transactions' table for filesys
tem /var/subversion/distribution/linux-x86/db:
Cannot allocate memory
(Perhaps this is related to issue 1224 - "svn update crashes with "Cannot
allocate memory""). If I change Apache's "MaxClients 150" to "MaxClients 50" my
perl program slows down massively but runs to completion (similarly if I insert
sleeps into the code it runs to completion).
If I run this perl directly on the machine with a local URL
./x.pl -u file///var/subversion/distribution/linux-x86
I get:
Couldn't open a repository: Unable to open an ra_local session to URL: Unable
to open repository
'file:///var/subversion/distribution/linux-x86/common/libraries/extras/tags/rele
ase-1_0_32/libraries/extras': Berkeley DB error while opening 'nodes' table for
filesystem /var/subversion/distribution/linux-x86/db:
Cannot allocate memory at ./x.pl line 11
My repository appears to wedge after the later (perhaps issue 739 - "automatic
db_recovery feature in libsvn_fs").
This is really worrying for us. We've been moving our source control to
subversion for the last month and now it seems we can wedge the repository
very easily and lock everyone out of it.
The machine we are using has 2Gb of memory and 30Gb of disk and the repository
is not that large (6Mb) but it contains a lot of external references to other
repositories totalling around 0.5Gb.
Any one got any suggestions?
The Perl is:
#!/usr/local/bin/perl
use strict ;
use Getopt::Std ;
use SVN::Client;
use Data::Dumper;
sub process_dir
{
my ( $ctx, $url, $revision, $spaces ) = @_ ;
my $dirs = $ctx->ls ( $url , $revision, 0 ) ;
foreach my $dir (sort keys (%$dirs)) {
my $printed = 0;
my $props = $ctx->propget("svn:externals", "$url/$dir", $revision, 0);
if (%$props + 0) {
print " " x $spaces, $dir, "\n";
$printed++;
foreach my $prop (keys %$props) {
my $y = $props->{$prop};
$y =~ tr/\r//d;
my @x = split /\s+/,$y;
for (my $i = 0; $i < $#x; $i+=2) {
print " " x $spaces, " ", $x[$i], " -> ", $x[$i+1],
"\n";
}
}
}
if ($dirs->{$dir}->kind() == $SVN::Node::dir) {
print " " x $spaces, $dir, "\n" if ($printed eq 0);
process_dir($ctx, "$url/$dir", $revision, $spaces+1);
}
elsif ($dirs->{$dir}->kind() == $SVN::Node::file) {
print " " x $spaces, "$dir\n" if ($printed eq 0);
} else {
print $dir, "\n";
exit 0;
}
}
}
my $url ;
my $ctx = new SVN::Client(
auth => [SVN::Client::get_simple_provider(),
SVN::Client::get_simple_prompt_provider(\&simple_prompt,2),
SVN::Client::get_username_provider()]
);
my $revision = 'HEAD' ;
my %options ;
getopts("u:",\%options);
if ( $options{'u'} ) { $url = $options{'u'}; }
if ( $options{'r'} ) { $revision = $options{'r'}; }
print "[$url]\n";
process_dir ( $ctx, $url, $revision, 0 )
Martin
--
Martin J. Evans
Easysoft Ltd, UK
Development
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
RE: svn client fails with Berkeley DB error Cannot allocate memory- repository wedges
Posted by James Goodall <jg...@dmetrix.com>.
I'm seeing a similar error, although under different conditions. Just after
upgrading to 1.0.4, I started getting hung
httpd->pre-commit->python2->svnperms.py processes and when enough of them
accrue (~10), I get this (or something similar) error message:
Berkeley DB error while opening 'uuids' table for filesystem
svn_repository/db:
Cannot allocate memory
I gather that the hung process are hogging all of the locks somehow, but I
have no idea why the procs don't complete. It seems to only happen during a
nightly script that automatically checks out files, adds copyright
information to them, and checks them back in. The script is slightly
misconfigured such that it tries to update files under /tags which
svnperms.py is configured to deny, if that makes a difference (repeatedly
failing a permission check for checkin causes the process to hang??).
Restarting apache seems to (temporarily) clear things up.
I did not see this problem at all under 1.0.2 and I haven't changed anything
else in the system since the svn upgrade.
SVN: 1.0.4 (ra_dav)
HTTPD: 2.0.49
svnperms.py: whatever is shipped in 1.0.4 tarball
python: 2.1.1
- James
-----Original Message-----
From: Ben Collins-Sussman [mailto:sussman@collab.net]
Sent: Sunday, May 30, 2004 7:55 PM
To: Martin J. Evans
Cc: users@subversion.tigris.org
Subject: Re: svn client fails with Berkeley DB error Cannot allocate memory-
repository wedges
On Fri, 2004-05-28 at 06:06, Martin J. Evans wrote:
> Berkeley DB error while opening 'transactions' table for filesys
> tem /var/subversion/distribution/linux-x86/db:
> Cannot allocate memory
IIRC, this error often means that BDB has run out of lock objects. Try
increasing the the number of locks in your repos/db/DB_CONFIG file and
then run 'svnadmin recover'.
I'd like to know what your script is doing (or what Web::SVN is doing)
that is so lock-intensive...
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: svn client fails with Berkeley DB error Cannot allocate memory - repository wedges
Posted by "Martin J. Evans" <ma...@easysoft.com>.
Ben,
On 31-May-2004 Ben Collins-Sussman wrote:
> On Fri, 2004-05-28 at 06:06, Martin J. Evans wrote:
>> Berkeley DB error while opening 'transactions' table for filesys
>> tem /var/subversion/distribution/linux-x86/db:
>> Cannot allocate memory
>
> IIRC, this error often means that BDB has run out of lock objects. Try
> increasing the the number of locks in your repos/db/DB_CONFIG file and
> then run 'svnadmin recover'.
Thanks for that. I take you are referring to:
set_lk_max_locks 2000
set_lk_max_lockers 2000
set_lk_max_objects 2000
Bumping all of these to 4000 did not help. When using file URL:
Couldn't open a repository: Unable to open an ra_local session to URL: Unable
to open repository
'file:///var/subversion/distribution/linux-x86/common/libraries/extras/tags':
Berkeley DB error while opening 'uuids' table for filesystem
/var/subversion/distribution/linux-x86/db:
Cannot allocate memory at /tmp/x.pl line 11
line 11 is an SVN::Client ls call.
Once I've run the script:
sh-2.05$ svnadmin recover linux-x86
Please wait; recovering the repository may take some time...
svn: DB_RUNRECOVERY: Fatal error, run database recovery
The last lines of strace show (I don't see the reason for the carp):
stat64("/var/subversion/distribution/linux-x86/db/nodes",
{st_mode=S_IFREG|0644, st_size=86016, ...}) = 0
stat64("/var/subversion/distribution/linux-x86/db/nodes",
{st_mode=S_IFREG|0644, st_size=86016, ...}) = 0
open("/var/subversion/distribution/linux-x86/db/nodes", O_RDWR|O_LARGEFILE) = 4
fcntl64(4, F_SETFD, FD_CLOEXEC) = 0
read(4, "\6\0\0\0$\364\0\0\0\0\0\0b1\5\0\t\0\0\0\0\20\0\0\0\t\0"..., 512) = 512
close(4) = 0
stat64("/var/subversion/distribution/linux-x86/db/nodes",
{st_mode=S_IFREG|0644, st_size=86016, ...}) = 0
open("/var/subversion/distribution/linux-x86/db/nodes", O_RDWR|O_LARGEFILE) = 4
fcntl64(4, F_SETFD, FD_CLOEXEC) = 0
fstat64(4, {st_mode=S_IFREG|0644, st_size=86016, ...}) = 0
brk(0x8354000) = 0x8354000
munmap(0x40640000, 16384) = 0
munmap(0x4048a000, 327680) = 0
munmap(0x404da000, 1466368) = 0
close(4) = 0
munmap(0x40448000, 270336) = 0
munmap(0x40018000, 8192) = 0
stat64("/usr/local/lib/perl5/5.8.2/i686-linux/Carp/Heavy.pmc", 0xbfffef5c) = -1
ENOENT (No such file or directory)
open("/usr/local/lib/perl5/5.8.2/i686-linux/Carp/Heavy.pm",
O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
stat64("/usr/local/lib/perl5/5.8.2/Carp/Heavy.pmc", 0xbfffef5c) = -1 ENOENT (No
such file or directory)
open("/usr/local/lib/perl5/5.8.2/Carp/Heavy.pm", O_RDONLY|O_LARGEFILE) = 4
brk(0x8355000) = 0x8355000
fstat64(4, {st_mode=S_IFREG|0444, st_size=5918, ...}) = 0
old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
0x40018000
read(4, "# Carp::Heavy uses some variable"..., 4096) = 4096
brk(0x8357000) = 0x8357000
brk(0x8358000) = 0x8358000
brk(0x8359000) = 0x8359000
brk(0x835a000) = 0x835a000
brk(0x835b000) = 0x835b000
brk(0x835c000) = 0x835c000
brk(0x835d000) = 0x835d000
brk(0x835e000) = 0x835e000
brk(0x835f000) = 0x835f000
brk(0x8360000) = 0x8360000
brk(0x8361000) = 0x8361000
brk(0x8362000) = 0x8362000
read(4, ";\n my $caller = caller($i);\n "..., 4096) = 1822
brk(0x8363000) = 0x8363000
brk(0x8364000) = 0x8364000
brk(0x8365000) = 0x8365000
brk(0x8366000) = 0x8366000
read(4, "", 4096) = 0
close(4) = 0
munmap(0x40018000, 4096) = 0
write(2, "Couldn\'t open a repository: Unab"..., 294) = 294
fcntl64(3, F_SETLKW, {type=F_UNLCK, whence=SEEK_SET, start=0, len=0}) = 0
close(3) = 0
munmap(0x40017000, 4096) = 0
_exit(255) = ?
> I'd like to know what your script is doing (or what Web::SVN is doing)
> that is so lock-intensive...
The script simply does an ls on the repository then walks down the files/dirs
recursively calling itself on dirs. However, for each file/dir it gets
svn:externals to see if any are defined. The idea was to produce a tree of the
repository because the repository has a large number of svn:externals
references. I can mail you seperately with the tree output which would give you
an idea of the repository structure. The script was in my last email.
BTW, all that is required is running the script - there is no one else
accessing the repository.
Let me know if these is any further info that could help.
Martin
--
Martin J. Evans
Easysoft Ltd, UK
Development
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: svn client fails with Berkeley DB error Cannot allocate memory
- repository wedges
Posted by Ben Collins-Sussman <su...@collab.net>.
On Fri, 2004-05-28 at 06:06, Martin J. Evans wrote:
> Berkeley DB error while opening 'transactions' table for filesys
> tem /var/subversion/distribution/linux-x86/db:
> Cannot allocate memory
IIRC, this error often means that BDB has run out of lock objects. Try
increasing the the number of locks in your repos/db/DB_CONFIG file and
then run 'svnadmin recover'.
I'd like to know what your script is doing (or what Web::SVN is doing)
that is so lock-intensive...
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
RE: svn client fails with Berkeley DB error Cannot allocate memory - repository wedges
Posted by "Martin J. Evans" <ma...@easysoft.com>.
I forgot to say this is subversion 1.0.2.
sh-2.05$ svn --version
svn, version 1.0.2 (r9423)
compiled May 11 2004, 14:43:24
Copyright (C) 2000-2004 CollabNet.
Subversion is open source software, see http://subversion.tigris.org/
This product includes software developed by CollabNet (http://www.Collab.Net/).
The following repository access (RA) modules are available:
* ra_dav : Module for accessing a repository via WebDAV (DeltaV) protocol.
- handles 'http' schema
* ra_local : Module for accessing a repository on local disk.
- handles 'file' schema
* ra_svn : Module for accessing a repository using the svn network protocol.
- handles 'svn' schema
Martin
--
Martin J. Evans
Easysoft Ltd, UK
Development
On 28-May-2004 Martin J. Evans wrote:
> Hi,
>
> Since no one has yet replied to my "tree view of repository including
> following
> external references" posting I put together a small perl script using
> WEB::SVN
> to do the job (see below). Running this program to a dav URL I see in the
> Apache log:
>
> ./x.pl -u http://a.b.c/linux-x86
>
> [Fri May 28 09:33:34 2004] [error] [client 192.168.0.78] Could not fetch
> resourc
> e information. [500, #0]
> [Fri May 28 09:33:34 2004] [error] [client 192.168.0.78] Could not open the
> requ
> ested SVN filesystem [500, #160029]
> [Fri May 28 09:33:34 2004] [error] [client 192.168.0.78] (17)File exists:
> Could
> not open the requested SVN filesystem [500, #160029]
> [Fri May 28 09:34:04 2004] [error] [client 192.168.0.78] (20014)Error string
> not
> specified yet: Berkeley DB error while opening 'transactions' table for
> filesys
> tem /var/subversion/distribution/linux-x86/db:
> Cannot allocate memory
>
> (Perhaps this is related to issue 1224 - "svn update crashes with "Cannot
> allocate memory""). If I change Apache's "MaxClients 150" to "MaxClients 50"
> my
> perl program slows down massively but runs to completion (similarly if I
> insert
> sleeps into the code it runs to completion).
>
> If I run this perl directly on the machine with a local URL
>
> ./x.pl -u file///var/subversion/distribution/linux-x86
>
> I get:
>
> Couldn't open a repository: Unable to open an ra_local session to URL: Unable
> to open repository
> 'file:///var/subversion/distribution/linux-x86/common/libraries/extras/tags/re
> le
> ase-1_0_32/libraries/extras': Berkeley DB error while opening 'nodes' table
> for
> filesystem /var/subversion/distribution/linux-x86/db:
> Cannot allocate memory at ./x.pl line 11
>
> My repository appears to wedge after the later (perhaps issue 739 -
> "automatic
> db_recovery feature in libsvn_fs").
>
> This is really worrying for us. We've been moving our source control to
> subversion for the last month and now it seems we can wedge the repository
> very easily and lock everyone out of it.
>
> The machine we are using has 2Gb of memory and 30Gb of disk and the
> repository
> is not that large (6Mb) but it contains a lot of external references to other
> repositories totalling around 0.5Gb.
>
> Any one got any suggestions?
>
> The Perl is:
>
>#!/usr/local/bin/perl
> use strict ;
> use Getopt::Std ;
> use SVN::Client;
> use Data::Dumper;
>
> sub process_dir
> {
> my ( $ctx, $url, $revision, $spaces ) = @_ ;
> my $dirs = $ctx->ls ( $url , $revision, 0 ) ;
>
> foreach my $dir (sort keys (%$dirs)) {
> my $printed = 0;
> my $props = $ctx->propget("svn:externals", "$url/$dir", $revision,
> 0);
> if (%$props + 0) {
> print " " x $spaces, $dir, "\n";
> $printed++;
> foreach my $prop (keys %$props) {
> my $y = $props->{$prop};
> $y =~ tr/\r//d;
> my @x = split /\s+/,$y;
> for (my $i = 0; $i < $#x; $i+=2) {
> print " " x $spaces, " ", $x[$i], " -> ", $x[$i+1],
> "\n";
> }
> }
> }
> if ($dirs->{$dir}->kind() == $SVN::Node::dir) {
> print " " x $spaces, $dir, "\n" if ($printed eq 0);
> process_dir($ctx, "$url/$dir", $revision, $spaces+1);
> }
> elsif ($dirs->{$dir}->kind() == $SVN::Node::file) {
> print " " x $spaces, "$dir\n" if ($printed eq 0);
> } else {
> print $dir, "\n";
> exit 0;
> }
> }
> }
>
> my $url ;
> my $ctx = new SVN::Client(
> auth => [SVN::Client::get_simple_provider(),
> SVN::Client::get_simple_prompt_provider(\&simple_prompt,2),
> SVN::Client::get_username_provider()]
> );
> my $revision = 'HEAD' ;
> my %options ;
> getopts("u:",\%options);
>
> if ( $options{'u'} ) { $url = $options{'u'}; }
> if ( $options{'r'} ) { $revision = $options{'r'}; }
> print "[$url]\n";
> process_dir ( $ctx, $url, $revision, 0 )
>
> Martin
> --
> Martin J. Evans
> Easysoft Ltd, UK
> Development
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org