You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by Jay Cornwall <ja...@esuna.co.uk> on 2004/10/17 17:43:03 UTC
Repository instability
Hi,
I have a Subversion repository set up that seems to become corrupt with
alarming regularity. When this happens, the web interface gives this error:
"Could not open the requested SVN filesystem"
Attempting to recover the repository with the svnadmin fails, giving
this error:
"Repository lock acquired.
Please wait; recovering the repository may take some time...
svn: DB_RUNRECOVERY: Fatal error, run database recovery
svn: bdb: Logging region out of memory; you may need to increase its size
svn: bdb: Recovery function for LSN 81 689286 failed on backward pass
svn: bdb: PANIC: Cannot allocate memory
svn: bdb: PANIC: fatal region error detected; run recovery"
A further attempt to recover the repository with the command 'db_recover
-vech /var/svn/repos/db' gives the error:
"db_recover: Finding last valid log LSN: file: 81 offset 747054
db_recover: Recovery starting from [79][28]
db_recover: Logging region out of memory; you may need to increase its size
db_recover: Recovery function for LSN 81 690040 failed on backward pass
db_recover: PANIC: Cannot allocate memory
db_recover: PANIC: fatal region error detected; run recovery"
The repository itself is quite small and subject to very low user
interaction. The interfaces I have available are:
file:/// - Local access
http:// - Apache, read-only access
https:// - Apache, read/write access
The repository doesn't need any commits to be made in order to become
corrupted, and I know local access hasn't been used between repository
corruptions.
I did have some trouble a while back with Apache running as umask 022,
causing similar problems; I've now rectified this and ensured that all
DB files are group-writable even after corruption.
The only regular activity which occurs in my repository is a nightly
'svnadmin hotcopy', again with the correct umask set. Could this be
causing the repository to corrupt itself?
Software running:
Subversion 1.1.0
DB 4.2.52
Apache 2.0.52
Any ideas?
--
Jay
http://www.esuna.co.uk/ - Coming soon, community blogging at its best!
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Repository instability
Posted by Jay Cornwall <ja...@esuna.co.uk>.
J Robert Ray wrote:
> I was getting a very similar error message to yours and fixed it by
> adjusting DB_CONFIG to the above values. After changing DB_CONFIG, the
> recover succeeded. For me, even when I was getting this error message
> from recover, the repository was not wedged and functioned normally. YMMV.
Thanks for yours (and everyone's) replies. :)
I've converted my repository to FSFS, so I'm not able to test if this
fixes the BDB problem at the moment. The good news is that the
repository does seem stable so far, so I'll stick with this backend for now.
If I get a chance, I'll try some of the suggestions made to see what was
causing the problems with the BDB backend.
(and apologies for being a bit late in replying, had some work to take
care of the last couple of days)
--
Jay
http://www.esuna.co.uk/ - Coming soon, community blogging at its best!
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Repository instability
Posted by J Robert Ray <jr...@imageworks.com>.
Jay Cornwall wrote:
> Hi,
>
> I have a Subversion repository set up that seems to become corrupt with
> alarming regularity. When this happens, the web interface gives this error:
>
> "Could not open the requested SVN filesystem"
>
> Attempting to recover the repository with the svnadmin fails, giving
> this error:
>
> "Repository lock acquired.
> Please wait; recovering the repository may take some time...
> svn: DB_RUNRECOVERY: Fatal error, run database recovery
> svn: bdb: Logging region out of memory; you may need to increase its size
> svn: bdb: Recovery function for LSN 81 689286 failed on backward pass
> svn: bdb: PANIC: Cannot allocate memory
> svn: bdb: PANIC: fatal region error detected; run recovery"
From an earlier post:
"""
Repository lock acquired.
Please wait; recovering the repository may take some time...
svn: DB_RUNRECOVERY: Fatal error, run database recovery
svn: bdb: Logging region out of memory; you may need to increase its size
svn: bdb: Recovery function for LSN 17571 340797 failed on backward pass
svn: bdb: PANIC: Cannot allocate memory
svn: bdb: PANIC: fatal region error detected; run recovery
svn: bdb: PANIC: fatal region error detected; run recovery
...
In a different test repository when I had this problem I bumped up the
DB_CONFIG values to look like this:
set_cachesize 0 8388608 8
set_lg_regionmax 524288
set_lg_bsize 2097152
set_lg_max 8388608
"""
I was getting a very similar error message to yours and fixed it by
adjusting DB_CONFIG to the above values. After changing DB_CONFIG, the
recover succeeded. For me, even when I was getting this error message
from recover, the repository was not wedged and functioned normally. YMMV.
- Robert
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Repository instability
Posted by Oliver Jowett <ol...@opencloud.com>.
Jay Cornwall wrote:
> Eric Gillespie wrote:
>
>> I love it when recover tells you to run recovery. When that
>> happens, i restored from backups. Before switching to fsfs, we
>> had a recoverable wedge about once a month, and one of these
>> non-recoverable wedges about once every other month. If
>> catastrophic recovery fails, i don't know that there is another
>> way to recover. I just restored from backups at that point.
>
>
> I've had to restore from backups many times over the last few months.
> Thankfully it's just an experimental repository; goodness only knows how
> much work I might have lost had I been using it for a live project.
We've had a much more positive experience with bdb. We've been running a
bdb repository via svnserve for the last 3 months. 600mb repository
across 28000 files, 13000 imported-from-cvs commits plus 700 new
commits, 10 developers, plenty of concurrent access going on.
It has never wedged or needed recovery.
-O
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Repository instability
Posted by John Szakmeister <jo...@szakmeister.net>.
On Sunday 17 October 2004 18:33, Jay Cornwall wrote:
> John Szakmeister wrote:
> > What version of BDB are you using? 4.1.x is known to have a shared
> > memory bug, that Subversion brings out in full color, especially on
> > machines with multiple processors. You should be using the latest
> > stable version, 4.2.52.
>
> I'm using BDB 4.2.52, as mentioned in my first post. The server is
> running on RedHat Enterprise Linux 3.0 ES which comes with BDB 4.1.x by
> default; I ensure Subversion uses the correct DB library (installed
> into /usr/local/db) on compilation, and I've checked that it (and
> Apache) link to the correct version with ldd.
>
> If it's possible I missed something else which might have to link to
> BDB 4.2, maybe that's the problem?
We upgraded to RHEL 3.0 ES about 6 months ago, and have experienced no
problems. Just to double check things, I would do an ldd on svn,
svnadmin, svnlook, httpd, and mod_dav_svn.so. All of them should have
db-4.2 listed and no other versions. If you see both db-4.1 and db-4.2,
then you've got a problem. Also, if you haven't done so, there are a
couple of patches on Sleepycat's webpage. You might want to apply those
to db-4.2.52.
And yes, if one of those isn't correctly linked to right DB library, you
can cause yourself some headaches, and potentially corrupt the
repository.
Also, getting the permissions right is difficult, especially when you
throw in file access. When you've found your repository horked, have you
ever noticed if the permissions where off? In my testing, I've found
that strange things can happen if the permissions were off, but I've
never seen the errors that you have. Is there some reason you need file
access? I've found that restricting access to just one method really
helps to avoid problems.
-John
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Repository instability
Posted by John Szakmeister <jo...@szakmeister.net>.
On Monday 18 October 2004 13:41, Eric Gillespie wrote:
> John Szakmeister <jo...@szakmeister.net> writes:
> > This is FUD.
>
> It is fact.
What's a fact it that's it's difficult to get right, and that it may
involve the use of wrapper scripts. What's a fact is that this sort of
thing is easier to setup with FSFS due to it's design. But BDB is able
to be used in the situation he has described.
> > BDB is just fine when the multiple access methods are set up
> > correctly.
>
> Incorrect. It is clear that Subversion's bdb problems are tied
> to usage patterns in some way, thus some people have trouble and
> some don't. But, there are problems even with correct usage.
I'll freely admit that the number of permissions issues that come up can
be overwhelming. But it doesn't preclude the use of BDB as the backend,
and particularly with.
> > With that said, I have Subversion setup for about 14 developers
> > and have used it for over 2 years with the BDB backend. Not
> > once have I run into any problems like this.
>
> 14, eh? Try 200.
Fantastic! I'm glad it's got such wide spread use at your facility. The
point was that it's being used every day and doesn't break.
-John
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Repository instability
Posted by Holger Hoffstätte <ho...@wizards.de>.
Jay Cornwall wrote:
> I'm using BDB 4.2.52, as mentioned in my first post. The server is
> running on RedHat Enterprise Linux 3.0 ES which comes with BDB 4.1.x by
> default; I ensure Subversion uses the correct DB library (installed into
> /usr/local/db) on compilation, and I've checked that it (and Apache)
> link to the correct version with ldd.
Judging from your error description it seems unlikely to be your fault.
Interestingly enough, the error message (about some memory page being
empty/corrupted) sounds suspiciously like something that pops up on the
PostgreSQL lists from time to time, where similar symptoms lead to
database corruption 'out of the blue'. As far as I can remember from those
threads the only common thing in all cases was that they happened on RH
Enterprise. This really sounds like some kernel/virtual memory/buffer
cache/VFS/ext3/driver problem where pages/blocks are not or not fully
written to disk. I know I occasionally had some of these under 2.4.x
(SuSE, not RH) when I still used it.
I really do not want to dis RH but personally I wouldn't touch their
transmogrified 2.4/2.6 Frankenkernel with the proverbial pitchfork (the
_regular_ kernel changelogs are scary enough). Have you considered trying
another kernel, e.g. vanilla 2.6.8? If that's not an option you could try
an fsfs repository when 1.1.1 comes out. After all it can't get worse.. :-}
Of course, all that being said, it's likely that in a few minutes someone
else will claim they've been running massive bdb repos on RHEL for ages
without any problems. All I know is that my company's >10GB repo has been
working flawlessly ever since it was put into production almost two years
ago, and survived migrations from RH7 and SuSE9 to Gentoo without any
hiccups at all.
Holger
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Repository instability
Posted by Jay Cornwall <ja...@esuna.co.uk>.
John Szakmeister wrote:
> What version of BDB are you using? 4.1.x is known to have a shared memory
> bug, that Subversion brings out in full color, especially on machines
> with multiple processors. You should be using the latest stable version,
> 4.2.52.
I'm using BDB 4.2.52, as mentioned in my first post. The server is
running on RedHat Enterprise Linux 3.0 ES which comes with BDB 4.1.x by
default; I ensure Subversion uses the correct DB library (installed into
/usr/local/db) on compilation, and I've checked that it (and Apache)
link to the correct version with ldd.
If it's possible I missed something else which might have to link to BDB
4.2, maybe that's the problem?
--
Jay
http://www.esuna.co.uk/ - Coming soon, community blogging at its best!
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Repository instability
Posted by Eric Gillespie <ep...@pretzelnet.org>.
John Szakmeister <jo...@szakmeister.net> writes:
> This is FUD.
It is fact.
> BDB is just fine when the multiple access methods are set up
> correctly.
Incorrect. It is clear that Subversion's bdb problems are tied
to usage patterns in some way, thus some people have trouble and
some don't. But, there are problems even with correct usage.
> With that said, I have Subversion setup for about 14 developers
> and have used it for over 2 years with the BDB backend. Not
> once have I run into any problems like this.
14, eh? Try 200.
--
Eric Gillespie <*> epg@pretzelnet.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Repository instability
Posted by John Szakmeister <jo...@szakmeister.net>.
On Sunday 17 October 2004 16:51, Jay Cornwall wrote:
> Eric Gillespie wrote:
> >>I have a Subversion repository set up that seems to become
> >>corrupt with alarming regularity.
> >
> > That's life with the berkeley back-end. I recommend switching to
> > fsfs.
This is FUD. I admit that FSFS is a bit easier to cope with, but BDB is
just fine when the multiple access methods are set up correctly.
> Thanks, I hadn't taken the time to read up on FSFS before. Seems fairly
> straightforward so I've dumped a backup of my repository into an FSFS
> repository; let's hope it performs a little better!
>
> >>db_recover: PANIC: fatal region error detected; run recovery"
> >
> > I love it when recover tells you to run recovery. When that
> > happens, i restored from backups. Before switching to fsfs, we
> > had a recoverable wedge about once a month, and one of these
> > non-recoverable wedges about once every other month. If
> > catastrophic recovery fails, i don't know that there is another
> > way to recover. I just restored from backups at that point.
>
> I've had to restore from backups many times over the last few months.
> Thankfully it's just an experimental repository; goodness only knows
> how much work I might have lost had I been using it for a live project.
What version of BDB are you using? 4.1.x is known to have a shared memory
bug, that Subversion brings out in full color, especially on machines
with multiple processors. You should be using the latest stable version,
4.2.52.
With that said, I have Subversion setup for about 14 developers and have
used it for over 2 years with the BDB backend. Not once have I run into
any problems like this.
-John
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Repository instability
Posted by Eric Gillespie <ep...@pretzelnet.org>.
Jay Cornwall <ja...@esuna.co.uk> writes:
> Thankfully it's just an experimental repository; goodness only knows how
> much work I might have lost had I been using it for a live project.
If you lose work, that's a failure of your backup/restore procedure.
--
Eric Gillespie <*> epg@pretzelnet.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Repository instability
Posted by Jay Cornwall <ja...@esuna.co.uk>.
Eric Gillespie wrote:
>>I have a Subversion repository set up that seems to become
>>corrupt with alarming regularity.
> That's life with the berkeley back-end. I recommend switching to
> fsfs.
Thanks, I hadn't taken the time to read up on FSFS before. Seems fairly
straightforward so I've dumped a backup of my repository into an FSFS
repository; let's hope it performs a little better!
>>db_recover: PANIC: fatal region error detected; run recovery"
> I love it when recover tells you to run recovery. When that
> happens, i restored from backups. Before switching to fsfs, we
> had a recoverable wedge about once a month, and one of these
> non-recoverable wedges about once every other month. If
> catastrophic recovery fails, i don't know that there is another
> way to recover. I just restored from backups at that point.
I've had to restore from backups many times over the last few months.
Thankfully it's just an experimental repository; goodness only knows how
much work I might have lost had I been using it for a live project.
--
Jay
http://www.esuna.co.uk/ - Coming soon, community blogging at its best!
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Repository instability
Posted by Eric Gillespie <ep...@pretzelnet.org>.
Jay Cornwall <ja...@esuna.co.uk> writes:
> I have a Subversion repository set up that seems to become
> corrupt with alarming regularity.
That's life with the berkeley back-end. I recommend switching to
fsfs.
> A further attempt to recover the repository with the command 'db_recover
> -vech /var/svn/repos/db' gives the error:
[...]
> db_recover: PANIC: fatal region error detected; run recovery"
I love it when recover tells you to run recovery. When that
happens, i restored from backups. Before switching to fsfs, we
had a recoverable wedge about once a month, and one of these
non-recoverable wedges about once every other month. If
catastrophic recovery fails, i don't know that there is another
way to recover. I just restored from backups at that point.
--
Eric Gillespie <*> epg@pretzelnet.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org