You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by Alblas Huibert <Hu...@metro-cc.de> on 2004/06/01 08:28:00 UTC
AW: svn client fails with Berkeley DB error Cannot allocate memor
y - repository wedges
Hi,
If you call your script recursivly,
it could end up having _many_ "open connections"....
Locks cannot be released until the scripts ends,
and the script only ends when a subprocess is done.
I can imagine that 4000 is a limmit way to small on
a medium sized repository.
The way you describe your sript it should release
the subversion server after each "svn ls",
but maybe something is wrong here.
Exausting your resources is very simple using recursion :-)
Maybe you should rewrite your script to first create
an infomation file about which parts of the tree to access.
Then rocess this info afterwards,
thus separating information gathering, and information processing.
Huibert Alblas
--
To undestand recursion,
we must first understand recursion
-----Ursprüngliche Nachricht-----
Von: Martin J. Evans [mailto:martin.evans@easysoft.com]
Gesendet: Dienstag, 1. Juni 2004 10:10
An: users@subversion.tigris.org
Cc: sussman@collab.net
Betreff: Re: svn client fails with Berkeley DB error Cannot allocate memory
- repository wedges
Ben,
On 31-May-2004 Ben Collins-Sussman wrote:
> On Fri, 2004-05-28 at 06:06, Martin J. Evans wrote:
>> Berkeley DB error while opening 'transactions' table for filesys tem
>> /var/subversion/distribution/linux-x86/db:
>> Cannot allocate memory
>
> IIRC, this error often means that BDB has run out of lock objects.
> Try increasing the the number of locks in your repos/db/DB_CONFIG file
> and then run 'svnadmin recover'.
Thanks for that. I take you are referring to:
set_lk_max_locks 2000
set_lk_max_lockers 2000
set_lk_max_objects 2000
Bumping all of these to 4000 did not help. When using file URL:
Couldn't open a repository: Unable to open an ra_local session to URL:
Unable to open repository
'file:///var/subversion/distribution/linux-x86/common/libraries/extras/tags'
:
Berkeley DB error while opening 'uuids' table for filesystem
/var/subversion/distribution/linux-x86/db:
Cannot allocate memory at /tmp/x.pl line 11
line 11 is an SVN::Client ls call.
Once I've run the script:
sh-2.05$ svnadmin recover linux-x86
Please wait; recovering the repository may take some time...
svn: DB_RUNRECOVERY: Fatal error, run database recovery
The last lines of strace show (I don't see the reason for the carp):
stat64("/var/subversion/distribution/linux-x86/db/nodes",
{st_mode=S_IFREG|0644, st_size=86016, ...}) = 0
stat64("/var/subversion/distribution/linux-x86/db/nodes",
{st_mode=S_IFREG|0644, st_size=86016, ...}) = 0
open("/var/subversion/distribution/linux-x86/db/nodes", O_RDWR|O_LARGEFILE)
= 4
fcntl64(4, F_SETFD, FD_CLOEXEC) = 0
read(4, "\6\0\0\0$\364\0\0\0\0\0\0b1\5\0\t\0\0\0\0\20\0\0\0\t\0"..., 512) =
512
close(4) = 0
stat64("/var/subversion/distribution/linux-x86/db/nodes",
{st_mode=S_IFREG|0644, st_size=86016, ...}) = 0
open("/var/subversion/distribution/linux-x86/db/nodes", O_RDWR|O_LARGEFILE)
= 4
fcntl64(4, F_SETFD, FD_CLOEXEC) = 0
fstat64(4, {st_mode=S_IFREG|0644, st_size=86016, ...}) = 0
brk(0x8354000) = 0x8354000
munmap(0x40640000, 16384) = 0
munmap(0x4048a000, 327680) = 0
munmap(0x404da000, 1466368) = 0
close(4) = 0
munmap(0x40448000, 270336) = 0
munmap(0x40018000, 8192) = 0
stat64("/usr/local/lib/perl5/5.8.2/i686-linux/Carp/Heavy.pmc", 0xbfffef5c) =
-1 ENOENT (No such file or directory)
open("/usr/local/lib/perl5/5.8.2/i686-linux/Carp/Heavy.pm",
O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
stat64("/usr/local/lib/perl5/5.8.2/Carp/Heavy.pmc", 0xbfffef5c) = -1 ENOENT
(No such file or directory) open("/usr/local/lib/perl5/5.8.2/Carp/Heavy.pm",
O_RDONLY|O_LARGEFILE) = 4
brk(0x8355000) = 0x8355000
fstat64(4, {st_mode=S_IFREG|0444, st_size=5918, ...}) = 0 old_mmap(NULL,
4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40018000
read(4, "# Carp::Heavy uses some variable"..., 4096) = 4096
brk(0x8357000) = 0x8357000
brk(0x8358000) = 0x8358000
brk(0x8359000) = 0x8359000
brk(0x835a000) = 0x835a000
brk(0x835b000) = 0x835b000
brk(0x835c000) = 0x835c000
brk(0x835d000) = 0x835d000
brk(0x835e000) = 0x835e000
brk(0x835f000) = 0x835f000
brk(0x8360000) = 0x8360000
brk(0x8361000) = 0x8361000
brk(0x8362000) = 0x8362000
read(4, ";\n my $caller = caller($i);\n "..., 4096) = 1822
brk(0x8363000) = 0x8363000
brk(0x8364000) = 0x8364000
brk(0x8365000) = 0x8365000
brk(0x8366000) = 0x8366000
read(4, "", 4096) = 0
close(4) = 0
munmap(0x40018000, 4096) = 0
write(2, "Couldn\'t open a repository: Unab"..., 294) = 294 fcntl64(3,
F_SETLKW, {type=F_UNLCK, whence=SEEK_SET, start=0, len=0}) = 0
close(3) = 0
munmap(0x40017000, 4096) = 0
_exit(255) = ?
> I'd like to know what your script is doing (or what Web::SVN is doing)
> that is so lock-intensive...
The script simply does an ls on the repository then walks down the
files/dirs recursively calling itself on dirs. However, for each file/dir it
gets svn:externals to see if any are defined. The idea was to produce a tree
of the repository because the repository has a large number of svn:externals
references. I can mail you seperately with the tree output which would give
you an idea of the repository structure. The script was in my last email.
BTW, all that is required is running the script - there is no one else
accessing the repository.
Let me know if these is any further info that could help.
Martin
--
Martin J. Evans
Easysoft Ltd, UK
Development
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: svn client fails with Berkeley DB error Cannot allocate memor y - repository wedges
Posted by "Martin J. Evans" <ma...@easysoft.com>.
I basically agree that an ls operation should not need that many locks. We only
got to a discussion of locks because Ben Collins-Sussman thought the error
Berkeley DB error while opening 'transactions' table for filesystem
/var/subversion/distribution/linux-x86/db:
Cannot allocate memory
might be related to running out of BDB object locks (and in deed it may still
be).
I tried doubling the locks in DB_CONFIG but it made no difference. However
since that repository does not recover (fails saying I need to run recover)
that test is probably meaningless. I've tried again with a completely new
repository that is working and changing the lock limits in DB_CONFIG (doubled
them to 4000). That did not help - the perl script fails in exactly the same
place.
You don't say if you were doing an ls() with recurse enabled - I am not. I am
doing ls() with no recurse and recursing myself in my perl (i.e. doing another
ls).
Anyway, I can replicate the problem at will now by doing this (admittedly this
is with subversion 1.0.2 but I cannot upgrade the server today, BDB is 4.2.52):
1. get hold of subversion sources
cd /tmp
svn co http://svn.collab.net/repos/svn/trunk subversion
cd subversion
find . -name .svn -exec rm -fr {} \;
2. create test repository
cd dir where repositories create (/var/subversion/distribution for me).
svnadmin create test-martin
3. import subversion tree into test repository.
cd /tmp/subversion
svn import . file:///var/subversion/distribution/test-martin -m "test"
4. prove the db is OK
cd /var/subversion/distribution
svnadmin recover test-martin
sh-2.05$ svnadmin recover test-martin/
Please wait; recovering the repository may take some time...
Recovery completed.
The latest repos revision is 1.
5. run the attached perl program with:
/tmp/x.pl -u file:///var/subversion/distribution/test-martin
Couldn't open a repository: Unable to open an ra_local session to URL: Unable
to open repository
'file:///var/subversion/distribution/test-martin/doc/book/README': Berkeley DB
error while opening 'uuids' table for filesystem
/var/subversion/distribution/test-martin/db:
Cannot allocate memory at /tmp/x.pl line 17
6. demonstrate DB is now broken and unrecoverable
cd /var/subversion/distribution
svnadmin recover test-martin
sh-2.05$ svnadmin recover test-martin
Please wait; recovering the repository may take some time...
svn: DB_RUNRECOVERY: Fatal error, run database recovery
This is repeatable at will for me.
I've followed the excellent hint from C. Michael Pilato and used db_stat before
and after running the perl script. The results were:
before:
2000 Maximum number of locks possible.
2000 Maximum number of lockers possible.
2000 Maximum number of lock objects possible.
9 Maximum number of locks at any one time.
17 Maximum number of lockers at any one time.
9 Maximum number of lock objects at any one time.
after:
2000 Maximum number of locks possible.
2000 Maximum number of lockers possible.
2000 Maximum number of lock objects possible.
883 Maximum number of locks at any one time.
1760 Maximum number of lockers at any one time.
19 Maximum number of lock objects at any one time.
I have never managed to recover a repository broken in this way but am open to
suggestions. All the arguments for changing the attached perl script to make it
succeed are irrelevant to me - I can make the script work fine via http by
reducing the number of max_client in apache or by introducing sleeps into it
(speed was never important for it). The error appears to happen at the
server end and it leaves my repository broken.
What I particularly don't like about it is that I can make this
happen using a http URL which implies to me I could break other public
repositories and they could break my public repositories. Needless to say I've
not tried this.
Martin
--
Martin J. Evans
Easysoft Ltd, UK
Development
On 01-Jun-2004 C. Michael Pilato wrote:
> "Martin J. Evans" <ma...@easysoft.com> writes:
>> On 01-Jun-2004 Alblas Huibert wrote:
>> > Locks cannot be released until the scripts ends,
>> > and the script only ends when a subprocess is done.
>> > I can imagine that 4000 is a limmit way to small on
>> > a medium sized repository.
>>
>> Fair enough but at most I would expect a read lock per ls and I
>> don't see why that cannot be released when the ls completes. Since
>> the depth of recursion I get to is only 9 I don't have any more than
>> 9 ls's going at once.
>
> Locks aren't taken out per-Subversion-operation. Their granularity is
> more tightly bound to Berkeley DB implementation details, something
> like per-row or per-page or something. That said, these locks don't
> outlive a Berkeley DB transaction, and there is usually a many-to-1
> relationship between Berkeley DB transactions and Subversion
> operations.
>
> I said all that to say that an 'ls' shouldn't eat *that* many locks.
>
> You can see how many locks a particular operation consumes (roughly)
> by doing this:
>
> svnadmin recover /path/to/repos
> (do the operation)
> db_stat -ch /path/to/repos/db | grep Maximum
>
> For example, I'm seeing about 31 locks used for a recursive 'ls' of
> the entire Subversion repository's /trunk directory.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
> For additional commands, e-mail: users-help@subversion.tigris.org
Re: svn client fails with Berkeley DB error Cannot allocate memor y - repository wedges
Posted by "C. Michael Pilato" <cm...@collab.net>.
"Martin J. Evans" <ma...@easysoft.com> writes:
> > Locks cannot be released until the scripts ends,
> > and the script only ends when a subprocess is done.
> > I can imagine that 4000 is a limmit way to small on
> > a medium sized repository.
>
> Fair enough but at most I would expect a read lock per ls and I
> don't see why that cannot be released when the ls completes. Since
> the depth of recursion I get to is only 9 I don't have any more than
> 9 ls's going at once.
Locks aren't taken out per-Subversion-operation. Their granularity is
more tightly bound to Berkeley DB implementation details, something
like per-row or per-page or something. That said, these locks don't
outlive a Berkeley DB transaction, and there is usually a many-to-1
relationship between Berkeley DB transactions and Subversion
operations.
I said all that to say that an 'ls' shouldn't eat *that* many locks.
You can see how many locks a particular operation consumes (roughly)
by doing this:
svnadmin recover /path/to/repos
(do the operation)
db_stat -ch /path/to/repos/db | grep Maximum
For example, I'm seeing about 31 locks used for a recursive 'ls' of
the entire Subversion repository's /trunk directory.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
RE: svn client fails with Berkeley DB error Cannot allocate memor y - repository wedges
Posted by "Martin J. Evans" <ma...@easysoft.com>.
On 01-Jun-2004 Alblas Huibert wrote:
> Hi,
>
> If you call your script recursivly,
> it could end up having _many_ "open connections"....
Agreed but the recursion depth only ever gets to 9 (thats not much). The actual
new SVN::Client is only done once and one ls in the recursive function.
> Locks cannot be released until the scripts ends,
> and the script only ends when a subprocess is done.
> I can imagine that 4000 is a limmit way to small on
> a medium sized repository.
Fair enough but at most I would expect a read lock per ls and I don't see why
that cannot be released when the ls completes. Since the depth of recursion I
get to is only 9 I don't have any more than 9 ls's going at once.
> The way you describe your sript it should release
> the subversion server after each "svn ls",
> but maybe something is wrong here.
This is what I suspect. It may be an issue in SVN::Client or in the subversion
libraries.
This worries me alot. We have spent a great deal of effort moving to subversion
and I can "destroy" the repository extremely easily. After running this script
once or twice on a local URL svn recover simply fails with:
sh-2.05$ svnadmin recover linux-x86/
Please wait; recovering the repository may take some time...
svn: DB_RUNRECOVERY: Fatal error, run database recovery
and running the script again gives:
[file:///var/subversion/distribution/linux-x86]
Couldn't open a repository: Unable to open an ra_local session to URL: Unable
to open repository 'file:///var/subversion/distribution/linux-x86': Berkeley DB
error while opening 'nodes' table for filesystem
/var/subversion/distribution/linux-x86/db:
Cannot allocate memory at /tmp/x.pl line 11
sh-2.05$
immediately - that is on the first ls. Each run of svnadmin recover allow the
script to run again up to the usual failure point.
> Exausting your resources is very simple using recursion :-)
>
> Maybe you should rewrite your script to first create
> an infomation file about which parts of the tree to access.
> Then rocess this info afterwards,
> thus separating information gathering, and information processing.
Yes we could do that. The script itself is not that important to us - we can
run it using an http URL and drop the apache max clients to 50 and it runs
slower but works. My concern is not making the script work but that something
is seriously going wrong with our repository. I'm not saying someone would
maliciously run this script to our subversion server but if they did it takes
the repository down and (currently) we've found no way to recover it. We do
want to use SVN::Client quite extensively.
At present our repository is broken. This is not the end of the world as I was
using a test repository (thank god).
> Huibert Alblas
>
> --
> To undestand recursion,
> we must first understand recursion
>
>
Martin
--
Martin J. Evans
Easysoft Ltd, UK
Development
> -----Urspr�ngliche Nachricht-----
> Von: Martin J. Evans [mailto:martin.evans@easysoft.com]
> Gesendet: Dienstag, 1. Juni 2004 10:10
> An: users@subversion.tigris.org
> Cc: sussman@collab.net
> Betreff: Re: svn client fails with Berkeley DB error Cannot allocate memory
> - repository wedges
>
>
> Ben,
>
> On 31-May-2004 Ben Collins-Sussman wrote:
>> On Fri, 2004-05-28 at 06:06, Martin J. Evans wrote:
>>> Berkeley DB error while opening 'transactions' table for filesys tem
>>> /var/subversion/distribution/linux-x86/db:
>>> Cannot allocate memory
>>
>> IIRC, this error often means that BDB has run out of lock objects.
>> Try increasing the the number of locks in your repos/db/DB_CONFIG file
>> and then run 'svnadmin recover'.
>
> Thanks for that. I take you are referring to:
>
> set_lk_max_locks 2000
> set_lk_max_lockers 2000
> set_lk_max_objects 2000
>
> Bumping all of these to 4000 did not help. When using file URL:
>
> Couldn't open a repository: Unable to open an ra_local session to URL:
> Unable to open repository
> 'file:///var/subversion/distribution/linux-x86/common/libraries/extras/tags'
>:
> Berkeley DB error while opening 'uuids' table for filesystem
> /var/subversion/distribution/linux-x86/db:
> Cannot allocate memory at /tmp/x.pl line 11
>
> line 11 is an SVN::Client ls call.
>
> Once I've run the script:
>
> sh-2.05$ svnadmin recover linux-x86
> Please wait; recovering the repository may take some time...
> svn: DB_RUNRECOVERY: Fatal error, run database recovery
>
> The last lines of strace show (I don't see the reason for the carp):
>
> stat64("/var/subversion/distribution/linux-x86/db/nodes",
> {st_mode=S_IFREG|0644, st_size=86016, ...}) = 0
> stat64("/var/subversion/distribution/linux-x86/db/nodes",
> {st_mode=S_IFREG|0644, st_size=86016, ...}) = 0
> open("/var/subversion/distribution/linux-x86/db/nodes", O_RDWR|O_LARGEFILE)
> = 4
> fcntl64(4, F_SETFD, FD_CLOEXEC) = 0
> read(4, "\6\0\0\0$\364\0\0\0\0\0\0b1\5\0\t\0\0\0\0\20\0\0\0\t\0"..., 512) =
> 512
> close(4) = 0
> stat64("/var/subversion/distribution/linux-x86/db/nodes",
> {st_mode=S_IFREG|0644, st_size=86016, ...}) = 0
> open("/var/subversion/distribution/linux-x86/db/nodes", O_RDWR|O_LARGEFILE)
> = 4
> fcntl64(4, F_SETFD, FD_CLOEXEC) = 0
> fstat64(4, {st_mode=S_IFREG|0644, st_size=86016, ...}) = 0
> brk(0x8354000) = 0x8354000
> munmap(0x40640000, 16384) = 0
> munmap(0x4048a000, 327680) = 0
> munmap(0x404da000, 1466368) = 0
> close(4) = 0
> munmap(0x40448000, 270336) = 0
> munmap(0x40018000, 8192) = 0
> stat64("/usr/local/lib/perl5/5.8.2/i686-linux/Carp/Heavy.pmc", 0xbfffef5c) =
> -1 ENOENT (No such file or directory)
> open("/usr/local/lib/perl5/5.8.2/i686-linux/Carp/Heavy.pm",
> O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
> stat64("/usr/local/lib/perl5/5.8.2/Carp/Heavy.pmc", 0xbfffef5c) = -1 ENOENT
> (No such file or directory) open("/usr/local/lib/perl5/5.8.2/Carp/Heavy.pm",
> O_RDONLY|O_LARGEFILE) = 4
> brk(0x8355000) = 0x8355000
> fstat64(4, {st_mode=S_IFREG|0444, st_size=5918, ...}) = 0 old_mmap(NULL,
> 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40018000
> read(4, "# Carp::Heavy uses some variable"..., 4096) = 4096
> brk(0x8357000) = 0x8357000
> brk(0x8358000) = 0x8358000
> brk(0x8359000) = 0x8359000
> brk(0x835a000) = 0x835a000
> brk(0x835b000) = 0x835b000
> brk(0x835c000) = 0x835c000
> brk(0x835d000) = 0x835d000
> brk(0x835e000) = 0x835e000
> brk(0x835f000) = 0x835f000
> brk(0x8360000) = 0x8360000
> brk(0x8361000) = 0x8361000
> brk(0x8362000) = 0x8362000
> read(4, ";\n my $caller = caller($i);\n "..., 4096) = 1822
> brk(0x8363000) = 0x8363000
> brk(0x8364000) = 0x8364000
> brk(0x8365000) = 0x8365000
> brk(0x8366000) = 0x8366000
> read(4, "", 4096) = 0
> close(4) = 0
> munmap(0x40018000, 4096) = 0
> write(2, "Couldn\'t open a repository: Unab"..., 294) = 294 fcntl64(3,
> F_SETLKW, {type=F_UNLCK, whence=SEEK_SET, start=0, len=0}) = 0
> close(3) = 0
> munmap(0x40017000, 4096) = 0
> _exit(255) = ?
>
>
>> I'd like to know what your script is doing (or what Web::SVN is doing)
>> that is so lock-intensive...
>
> The script simply does an ls on the repository then walks down the
> files/dirs recursively calling itself on dirs. However, for each file/dir it
> gets svn:externals to see if any are defined. The idea was to produce a tree
> of the repository because the repository has a large number of svn:externals
> references. I can mail you seperately with the tree output which would give
> you an idea of the repository structure. The script was in my last email.
>
> BTW, all that is required is running the script - there is no one else
> accessing the repository.
>
> Let me know if these is any further info that could help.
>
> Martin
> --
> Martin J. Evans
> Easysoft Ltd, UK
> Development
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org