You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by "Andreas J. Koenig" <an...@anima.de> on 2002/12/08 13:47:27 UTC
Story of BerkeleyDB failing
Executive summary:
==================
"svn log" on a large repository fails with "Cannot allocate memory". I
have no workaround. Can somebody provide advice?
Full story:
===========
'svn log' emited:
-----------------
Berkeley DB error while reading node revision for filesystem
/usr/local/svn/perl/db: Cannot allocate memory
--or--
svn: Berkeley DB error while reading transaction for filesystem
/usr/local/svn/perl/db: Cannot allocate memory
Failing commands were:
----------------------
svn log -r HEAD http://...
asked for password, then quickly failed; error_log of apache
logged the same error
svn log -r HEAD file://...
fails quickly. See below for an strace.
Commands that still worked:
---------------------------
svnadmin recover # didn't have to recover anything
svnadmin dump # fortunately!
svn st -u
svn log -r 1
My environment:
---------------
- svn's own revision was 3953. I later upgraded to 4045 with same
results.
- Berkeley version is 4.0.14.
- Linux Kernel is 2.5.47-ac5, LARGE_FILE support is available and
seems to be working. Later I switched to 2.5.50 and 2.4.20, same
results.
- In the beginning I didn't touch DB_CONFIG. Later I did try to lower
and increase values for set_lg_bsize and set_lg_max to no avail.
- The machine has 1.5 GB memory and 8 GB swap. The partition has 12 GB
total, at that point 3 GB were unused. The process size is moderate
(top reported 24.6 MB for SIZE immediately before the error
happened)
- I had managed to write 7333 Berkeley log.NNNNNNNNNN files, the last
revision was #13069. Total size of repository incl. logfiles was
~8GB.
- This error occured for the first time around revision 10600 but only
for a single command (do not know which command it was, most
probably 'log'). After that, Subversion continued to accept checkins
and allowed growing to >13000 revisions.
- removing logfiles did not change the situation
- the dump file was > 3GB. Fortunately I could spot a checkpoint that
seemed appropriate to recover a valuable part of the repository (rev
10408).
What I tried as a rescue:
-------------------------
With the dump (up to rev 10408; 2.3 GB) in my hands I removed the
whole repository, created a new one and successfully loaded the dump
into it. The repository could then be used to check out a working
copy, but it still could not send a log. It failed with the same
"Cannot allocate memory" from BerkeleyDB as before.
Limits:
-------
Maybe I have a limit in the shell environment? The limit command told
me:
cputime unlimited
filesize unlimited
datasize unlimited
stacksize 8MB
coredumpsize 0kB
memoryuse unlimited
maxproc 12287
descriptors 1024
memorylocked unlimited
addressspace unlimited
maxfilelocks unlimited
Doesn't look very limited. Increasing the stacksize up to 2048MB
didn't help.
Strace (v4.2) tells me:
pread(5, "\203\1\0\0\341\221\2\0\366\0\0\0\365\0\0\0\370\0\0\0\n"..., 4096, 1007
616) = 4096
brk(0x926d000) = 0x926d000
pread(5, "\200\1\0\0w\270\5\0\365\0\0\0\356\0\0\0\366\0\0\0\n\0l"..., 4096, 1003
520) = 4096
brk(0x9270000) = 0x9270000
brk(0x9279000) = 0x9279000
write(2, "subversion/libsvn_fs/err.c:47", 29subversion/libsvn_fs/err.c:47) = 29
write(2, ": (apr_err=160029, src_err=12)\n", 31: (apr_err=160029, src_err=12)
) = 31
write(2, "svn: Berkeley DB error\n", 23svn: Berkeley DB error
) = 23
write(2, "svn: Berkeley DB error while rea"..., 113svn: Berkeley DB error while
reading node revision for filesystem /usr/local/svn/perl/db:
Cannot allocate memory
) = 113
log -r 1 vs log -r HEAD
-----------------------
You might believe that if 'log -r 1' works and 'log -r HEAD' doesn't,
there would be a threshold that marks a region of revisions that works
and another region that doesn't. It turns out that this is the case,
but the threshold is not stable. At one point I determined the
threshold to be 4046/4047, then after a 'svnadmin recover' it was
9685/9686. After a while it was 9642/9643. It seems to depend on the
history of actions since the last recover. Indeed, whenever I run a
recover, the threshold becomes 9685/9686 again.
What I did not try:
-------------------
I haven't tried to upgrade to Berkeley 4.1.24.
Please advise what I should try next.--Thanks!
--
andreas
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Story of BerkeleyDB failing
Posted by Rafael Garcia-Suarez <rg...@free.fr>.
Andreas J. Koenig wrote:
> >>>>> On Sun, 08 Dec 2002 15:42:28 +0100, andreas.koenig@anima.de (Andreas J. Koenig) said:
>
> >>>>> On Sun, 8 Dec 2002 15:34:48 +0100, "Sander Striker" <st...@apache.org> said:
> >> Try doubling the values of the following options in DB_CONFIG
>
> >> set_lk_max_locks
> >> set_lk_max_lockers
> >> set_lk_max_objects
>
> > Doubling from 2000 to 4000 each. Same result. I had forgotten to
> > report that I tried that already.
>
> Uh-oh-wait a second: If I now run an 'svnadmin recover' then it works
> indeed.
>
> THANK YOU!
Anyway, is this a scalability problem ? I mean, will those thresholds
need to be increased when the repository this rev #20000 ? Why is this
necessary ? (will undeltifying the repository help here ?)
(And this ought to be mentioned in The Book, for those who don't know
how to administrate a huge Berkeley DB.)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: [PATCH] Re: Story of BerkeleyDB failing
Posted by Karl Fogel <kf...@newton.ch.collab.net>.
Rafael Garcia-Suarez <ra...@hexaflux.com> writes:
> Here's a small patch :
Thanks! Applied in revision 4067.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
[PATCH] Re: Story of BerkeleyDB failing
Posted by Rafael Garcia-Suarez <ra...@hexaflux.com>.
"Sander Striker" <st...@apache.org> wrote:
> > From: Branko Cibej [mailto:brane@xbc.nu]
> >>
> > Right, this should be a FAQ: Every time you change DB_CONFIG, you have
> > to run svnadmin recover, otherwise the changes don't take effect.
>
> We should put it in the comment lines in the prefabricated
> DB_CONFIG file we put in the repository.
Here's a small patch :
* subversion/libsvn_fs/fs.c
Add a comment in the default DB_CONFIG file about 'svnadmin recover'
needed whenever it's modified.
--- subversion/libsvn_fs/fs.c.orig Mon Dec 9 10:42:07 2002
+++ subversion/libsvn_fs/fs.c Mon Dec 9 10:44:16 2002
@@ -431,6 +431,8 @@ svn_fs_create_berkeley (svn_fs_t *fs, co
static const char dbconfig_contents[] =
"# This is the configuration file for the Berkeley DB environment\n"
"# used by your Subversion repository.\n"
+ "# You must run 'svnadmin recover' whenever you modify this file\n"
+ "# for your changes to take effect.\n"
"\n"
"### Lock subsystem\n"
"#\n"
End of Patch.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
RE: Story of BerkeleyDB failing
Posted by Sander Striker <st...@apache.org>.
> From: Branko Cibej [mailto:brane@xbc.nu]
> Sent: 08 December 2002 22:59
> Andreas J. Koenig wrote:
>>>> Try doubling the values of the following options in DB_CONFIG
>>
>>>> set_lk_max_locks
>>>> set_lk_max_lockers
>>>> set_lk_max_objects
>>
>>> Doubling from 2000 to 4000 each. Same result. I had forgotten to
>>> report that I tried that already.
>>
>> Uh-oh-wait a second: If I now run an 'svnadmin recover' then it works
>> indeed.
>>
> Right, this should be a FAQ: Every time you change DB_CONFIG, you have
> to run svnadmin recover, otherwise the changes don't take effect.
We should put it in the comment lines in the prefabricated
DB_CONFIG file we put in the repository.
Sander
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Story of BerkeleyDB failing
Posted by Branko Čibej <br...@xbc.nu>.
Andreas J. Koenig wrote:
>>>>>>On Sun, 08 Dec 2002 15:42:28 +0100, andreas.koenig@anima.de (Andreas J. Koenig) said:
>>>>>>
>>>>>>
>
>
>
>>>>>>On Sun, 8 Dec 2002 15:34:48 +0100, "Sander Striker" <st...@apache.org> said:
>>>>>>
>>>>>>
> >> Try doubling the values of the following options in DB_CONFIG
>
> >> set_lk_max_locks
> >> set_lk_max_lockers
> >> set_lk_max_objects
>
> > Doubling from 2000 to 4000 each. Same result. I had forgotten to
> > report that I tried that already.
>
>Uh-oh-wait a second: If I now run an 'svnadmin recover' then it works
>indeed.
>
Right, this should be a FAQ: Every time you change DB_CONFIG, you have
to run svnadmin recover, otherwise the changes don't take effect.
>THANK YOU!
>
--
Brane Čibej <br...@xbc.nu> http://www.xbc.nu/brane/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Story of BerkeleyDB failing
Posted by "Andreas J. Koenig" <an...@anima.de>.
>>>>> On Sun, 08 Dec 2002 15:42:28 +0100, andreas.koenig@anima.de (Andreas J. Koenig) said:
>>>>> On Sun, 8 Dec 2002 15:34:48 +0100, "Sander Striker" <st...@apache.org> said:
>> Try doubling the values of the following options in DB_CONFIG
>> set_lk_max_locks
>> set_lk_max_lockers
>> set_lk_max_objects
> Doubling from 2000 to 4000 each. Same result. I had forgotten to
> report that I tried that already.
Uh-oh-wait a second: If I now run an 'svnadmin recover' then it works
indeed.
THANK YOU!
--
andreas
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Story of BerkeleyDB failing
Posted by "Andreas J. Koenig" <an...@anima.de>.
>>>>> On Sun, 8 Dec 2002 15:34:48 +0100, "Sander Striker" <st...@apache.org> said:
> Try doubling the values of the following options in DB_CONFIG
> set_lk_max_locks
> set_lk_max_lockers
> set_lk_max_objects
Doubling from 2000 to 4000 each. Same result. I had forgotten to
report that I tried that already.
--
andreas
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
RE: Story of BerkeleyDB failing
Posted by Sander Striker <st...@apache.org>.
> From: k@k242.linux.bogus [mailto:k@k242.linux.bogus]On Behalf Of Andreas
> J. Koenig
> Sent: 08 December 2002 14:47
Hi,
> Executive summary:
> ==================
>
> "svn log" on a large repository fails with "Cannot allocate memory". I
> have no workaround. Can somebody provide advice?
[...]
> My environment:
> ---------------
>
> - svn's own revision was 3953. I later upgraded to 4045 with same
> results.
>
> - Berkeley version is 4.0.14.
>
> - Linux Kernel is 2.5.47-ac5, LARGE_FILE support is available and
> seems to be working. Later I switched to 2.5.50 and 2.4.20, same
> results.
>
> - In the beginning I didn't touch DB_CONFIG. Later I did try to lower
> and increase values for set_lg_bsize and set_lg_max to no avail.
[...]
> Please advise what I should try next.--Thanks!
Try doubling the values of the following options in DB_CONFIG
set_lk_max_locks
set_lk_max_lockers
set_lk_max_objects
Then get back at us. Thanks,
Sander
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org