You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by Frank Freund <ff...@memocomp.de> on 2004/07/01 22:34:19 UTC
Severe repository database corruption
Hi all,
today morning, and after my last commit yesterday (and noone accessed
the repository in any way in between) worked fine, I got the following
error message while trying to commit (using TortoiseSVN):
> "DB_RUNRECOVERY: Fatal error, run database recovery."
Checking the repository via the browser, I got: "Could not open the
requested SVN filesystem", so I started reading docs, mailing lists and
searched the web.
To anticipate, none of the things I tried worked :-( and it seems, that
the database is the problem (but maybe not the reason)
First I will give some information about the environment, then the
things I tried and finally some of the sources I found concerning (other
people having) this problem.
Environment
-------------------
The repository was running now for about half a year with nearly 400
commits and 700MB repository size.
The System configuration is:
SuSE 8.2 with
* Subversion 0.35.1 (now upgraded to 1.0.5, but the problem remains
exactly the same)
* BDB 4.2.52.1
* Apache 2.0.48
All above build from source, and the clientside
* Tortoise SVN 0.24.0 build 554
Attempts
-------------------
First I tried
> linux: # svnadmin recover /usr/local/apache2/svn
> Please wait; recovering the repository may take some time...
> Recovery completed.
> The latest repos revision is 394.
which looked fine, but didn't worked - same errors as above. So next was
> linux: # svnadmin verify /usr/local/apache2/svn
which also completed successfully, but didn't helped either. Next step
were the databasetools:
> linux: # ./db_recover -h /usr/local/apache2/svn/db
This worked, but I just could access the repository over the browser for
some tries, before the same error occurred again. (reproducable always)
Accessing the repository using TorsoiseSVN, I just tried checkouts,
failed directly (reproducable always)
Tried again:
> linux: # svnadmin verify /usr/local/apache2/svn
> svn: Berkeley DB error while opening environment for filesystem
/usr/local/apache2/svn/db:
> DB_RUNRECOVERY: Fatal error, run database recovery
and
> linux: # ./db_recover -h /usr/local/apache2/svn/db
which worked again for short (sometimes 10 seconds, never longer than
about 2 minutes). Some tries later the verify failed:
> linux: # svnadmin verify /usr/local/apache2/svn
> Verified revision 0.
> svn: Berkeley DB error while beginning Berkeley DB transaction for
filesystem /usr/local/apache2/svn/db:
> DB_RUNRECOVERY: Fatal error, run database recovery
> svn: Berkeley DB error while closing 'nodes' database for filesystem
/usr/local/apache2/svn/memocomp/db:
> DB_RUNRECOVERY: Fatal error, run database recovery
Meanwhile updated to SVN1.0.5 and imported the backup (Jesus saves:-)
successfully, but the symptoms remain the same.
Some posts mentioned problems with mixed up libdb's, but as far as I know a
> linux: # grep libdb /proc/17455/maps
> 4002a000-400e3000 r-xp 00000000 08:02 42429
/usr/local/BerkeleyDB.4.2/lib/libdb-4.2.so
> 400e3000-400e5000 rw-p 000b8000 08:02 42429
/usr/local/BerkeleyDB.4.2/lib/libdb-4.2.so
with 17455 beeing a httpd-PID, confirms me that this isn't the case.
The http-error log shows:
> [Thu Jul 01 08:11:02 2004] [error] [client 192.168.100.11] Could not
fetch resource information. [500, #0]
> [Thu Jul 01 08:11:02 2004] [error] [client 192.168.100.11] Could not
determine the proper revision to access [500, #160029]
> [Thu Jul 01 08:11:02 2004] [error] [client 192.168.100.11]
(13)Permission denied: Berkeley DB error while checkpointing after
Berkeley DB transaction for filesystem /usr/local/apache2/svn/db:
> DB_RUNRECOVERY: Fatal error, run database recovery [500, #160029]
> [Thu Jul 01 08:11:02 2004] [error] [client 192.168.100.11]
(20014)Error string not specified yet: Berkeley DB error while closing
'nodes' database for filesystem /usr/local/apache2/svn/db:
> DB_RUNRECOVERY: Fatal error, run database recovery
> [Thu Jul 01 08:17:02 2004] [error] [client 192.168.100.11]
(20014)Error string not specified yet: Berkeley DB error while opening
environment for filesystem /usr/local/apache2/svn/db:
> DB_RUNRECOVERY: Fatal error, run database recovery, referer:
http://192.168.100.2/repository.html
and later
> [Thu Jul 01 20:14:46 2004] [error] [client 192.168.100.11] Could not
fetch resource information. [500, #0], referer:
http://192.168.100.2/repository.html
> [Thu Jul 01 20:14:46 2004] [error] [client 192.168.100.11] Could not
open the requested SVN filesystem [500, #160029], referer:
http://192.168.100.2/repository.html
> [Thu Jul 01 20:14:46 2004] [error] [client 192.168.100.11] (17)File
exists: Could not open the requested SVN filesystem [500, #160029],
referer: http://192.168.100.2/repository.html
> [Thu Jul 01 20:14:47 2004] [error] [client 192.168.100.11]
(20014)Error string not specified yet: Berkeley DB error while opening
environment for filesystem /usr/local/apache2/svn/db:
> DB_RUNRECOVERY: Fatal error, run database recovery, referer:
http://192.168.100.2/repository.html
Further readings
-------------------
Repository recovery:
http://svnbook.red-bean.com/svnbook/ch05s03.html#svn-ch-5-sect-3.4
Another broken BDB:
http://www.contactor.se/~dast/svnusers/archive-2004-05/0556.shtml
Corrupted repository db:
http://www.contactor.se/~dast/svn/archive-2004-05/0071.shtml
Corrupted repository db:
http://www.contactor.se/~dast/svn/archive-2004-05/0097.shtml
Unrecoverable databases:
http://www.contactor.se/~dast/svnusers/archive-2004-05/1283.shtml
http://subversion.tigris.org/servlets/BrowseList?list=users&by=thread&from=202660
So that's all and I like to thank everyone who has read this &%&$*long post.
Thanks in advance for any help!
Frank
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Severe repository database corruption
Posted by Peter Poeml <po...@suse.de>.
On Fri, Jul 02, 2004 at 03:04:27PM +0200, Frank Freund wrote:
> Thanks *all* for the help, it seems that the permissions did the trick.
> I will give a final feedback next week.
I find the attached script helpful. It restores the permissions after
recovering.
> >-I see you doing a lot of stuff as root... be sure when your done to do
> >chmod -R user:group for who it should be owned by for the webserver or
> >synserver
Peter
--
Thought is limitation. Free your mind.
Re: Severe repository database corruption
Posted by Ben Collins-Sussman <su...@collab.net>.
On Fri, 2004-07-02 at 08:04, Frank Freund wrote:
> Thanks *all* for the help, it seems that the permissions did the trick.
> I will give a final feedback next week.
>
> But what I don't understand, and that's why I didn't digged in this
> direction yesterday is that the repository worked from its begin on, for
> about 6 months and 400 commits with around 700MB, with *exactly the same
> settings without any problem*.
> So how to explain this? What triggered the error?
The database will get "wedged" if any process accessing it suddenly
dies, as the FAQ explains. So perhaps some apache process crashed while
it had the database open. It could be something as simple as that:
nothing to do with your configuration or setup. The important thing to
learn, as an svn administrator, is that when database gets stuck, you
know how to un-stick it and keep permissions correct (because wrong
permissions will wedge a repository just as easily as a process crash.)
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Severe repository database corruption
Posted by Frank Freund <ff...@memocomp.de>.
Thanks *all* for the help, it seems that the permissions did the trick.
I will give a final feedback next week.
But what I don't understand, and that's why I didn't digged in this
direction yesterday is that the repository worked from its begin on, for
about 6 months and 400 commits with around 700MB, with *exactly the same
settings without any problem*.
So how to explain this? What triggered the error?
Frank
Aaron Wohl wrote:
>-I see you doing a lot of stuff as root... be sure when your done to do
>chmod -R user:group for who it should be owned by for the webserver or
>synserver
>
>-With the same sort of problems your seeing, I was able do svnadmin dump
>then svnadmin create and svnadmin load. Ill bet your stuff was
>recoverable without going to backup tapes.
>
>-during your work it looks like stuff was left as /usr/local/apache2/svn
>During recovery its important that noone else have access for stuff like
>this, I do mv /usr/local/apache2/svn /usr/local/apache2/svn.sick .
>Then do the repair work and put it back. Double check your
>owner/group/permission as that can make it look broken aftewards if root
>owns stuff.
>
>----- Original message -----
>From: "Frank Freund" <ff...@memocomp.de>
>To: users@subversion.tigris.org
>Date: Fri, 02 Jul 2004 00:34:19 +0200
>Subject: Severe repository database corruption
>
>Hi all,
>today morning, and after my last commit yesterday (and noone accessed
>the repository in any way in between) worked fine, I got the following
>error message while trying to commit (using TortoiseSVN):
> > "DB_RUNRECOVERY: Fatal error, run database recovery."
>Checking the repository via the browser, I got: "Could not open the
>requested SVN filesystem", so I started reading docs, mailing lists and
>searched the web.
>To anticipate, none of the things I tried worked :-( and it seems, that
>the database is the problem (but maybe not the reason)
>
>First I will give some information about the environment, then the
>things I tried and finally some of the sources I found concerning (other
>people having) this problem.
>
>Environment
>-------------------
>The repository was running now for about half a year with nearly 400
>commits and 700MB repository size.
>The System configuration is:
>SuSE 8.2 with
>* Subversion 0.35.1 (now upgraded to 1.0.5, but the problem remains
>exactly the same)
>* BDB 4.2.52.1
>* Apache 2.0.48
>All above build from source, and the clientside
>* Tortoise SVN 0.24.0 build 554
>
>Attempts
>-------------------
>First I tried
> > linux: # svnadmin recover /usr/local/apache2/svn
> > Please wait; recovering the repository may take some time...
> > Recovery completed.
> > The latest repos revision is 394.
>which looked fine, but didn't worked - same errors as above. So next was
> > linux: # svnadmin verify /usr/local/apache2/svn
>which also completed successfully, but didn't helped either. Next step
>were the databasetools:
> > linux: # ./db_recover -h /usr/local/apache2/svn/db
>This worked, but I just could access the repository over the browser for
>some tries, before the same error occurred again. (reproducable always)
>Accessing the repository using TorsoiseSVN, I just tried checkouts,
>failed directly (reproducable always)
>Tried again:
> > linux: # svnadmin verify /usr/local/apache2/svn
> > svn: Berkeley DB error while opening environment for filesystem
>/usr/local/apache2/svn/db:
> > DB_RUNRECOVERY: Fatal error, run database recovery
>and
> > linux: # ./db_recover -h /usr/local/apache2/svn/db
>which worked again for short (sometimes 10 seconds, never longer than
>about 2 minutes). Some tries later the verify failed:
> > linux: # svnadmin verify /usr/local/apache2/svn
> > Verified revision 0.
> > svn: Berkeley DB error while beginning Berkeley DB transaction for
>filesystem /usr/local/apache2/svn/db:
> > DB_RUNRECOVERY: Fatal error, run database recovery
> > svn: Berkeley DB error while closing 'nodes' database for filesystem
>/usr/local/apache2/svn/memocomp/db:
> > DB_RUNRECOVERY: Fatal error, run database recovery
>
>Meanwhile updated to SVN1.0.5 and imported the backup (Jesus saves:-)
>successfully, but the symptoms remain the same.
>Some posts mentioned problems with mixed up libdb's, but as far as I
>know a
> > linux: # grep libdb /proc/17455/maps
> > 4002a000-400e3000 r-xp 00000000 08:02 42429
>/usr/local/BerkeleyDB.4.2/lib/libdb-4.2.so
> > 400e3000-400e5000 rw-p 000b8000 08:02 42429
>/usr/local/BerkeleyDB.4.2/lib/libdb-4.2.so
>with 17455 beeing a httpd-PID, confirms me that this isn't the case.
>
>The http-error log shows:
> > [Thu Jul 01 08:11:02 2004] [error] [client 192.168.100.11] Could not
>fetch resource information. [500, #0]
> > [Thu Jul 01 08:11:02 2004] [error] [client 192.168.100.11] Could not
>determine the proper revision to access [500, #160029]
> > [Thu Jul 01 08:11:02 2004] [error] [client 192.168.100.11]
>(13)Permission denied: Berkeley DB error while checkpointing after
>Berkeley DB transaction for filesystem /usr/local/apache2/svn/db:
> > DB_RUNRECOVERY: Fatal error, run database recovery [500, #160029]
> > [Thu Jul 01 08:11:02 2004] [error] [client 192.168.100.11]
>(20014)Error string not specified yet: Berkeley DB error while closing
>'nodes' database for filesystem /usr/local/apache2/svn/db:
> > DB_RUNRECOVERY: Fatal error, run database recovery
> > [Thu Jul 01 08:17:02 2004] [error] [client 192.168.100.11]
>(20014)Error string not specified yet: Berkeley DB error while opening
>environment for filesystem /usr/local/apache2/svn/db:
> > DB_RUNRECOVERY: Fatal error, run database recovery, referer:
>http://192.168.100.2/repository.html
>and later
> > [Thu Jul 01 20:14:46 2004] [error] [client 192.168.100.11] Could not
>fetch resource information. [500, #0], referer:
>http://192.168.100.2/repository.html
> > [Thu Jul 01 20:14:46 2004] [error] [client 192.168.100.11] Could not
>open the requested SVN filesystem [500, #160029], referer:
>http://192.168.100.2/repository.html
> > [Thu Jul 01 20:14:46 2004] [error] [client 192.168.100.11] (17)File
>exists: Could not open the requested SVN filesystem [500, #160029],
>referer: http://192.168.100.2/repository.html
> > [Thu Jul 01 20:14:47 2004] [error] [client 192.168.100.11]
>(20014)Error string not specified yet: Berkeley DB error while opening
>environment for filesystem /usr/local/apache2/svn/db:
> > DB_RUNRECOVERY: Fatal error, run database recovery, referer:
>http://192.168.100.2/repository.html
>
>Further readings
>-------------------
>Repository recovery:
>http://svnbook.red-bean.com/svnbook/ch05s03.html#svn-ch-5-sect-3.4
>Another broken BDB:
>http://www.contactor.se/~dast/svnusers/archive-2004-05/0556.shtml
>Corrupted repository db:
>http://www.contactor.se/~dast/svn/archive-2004-05/0071.shtml
>Corrupted repository db:
>http://www.contactor.se/~dast/svn/archive-2004-05/0097.shtml
>Unrecoverable databases:
>http://www.contactor.se/~dast/svnusers/archive-2004-05/1283.shtml
>http://subversion.tigris.org/servlets/BrowseList?list=users&by=thread&from=202660
>
>So that's all and I like to thank everyone who has read this &%&$*long
>post.
>
>Thanks in advance for any help!
>Frank
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
>For additional commands, e-mail: users-help@subversion.tigris.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Severe repository database corruption
Posted by Aaron Wohl <su...@awohl.com>.
-I see you doing a lot of stuff as root... be sure when your done to do
chmod -R user:group for who it should be owned by for the webserver or
synserver
-With the same sort of problems your seeing, I was able do svnadmin dump
then svnadmin create and svnadmin load. Ill bet your stuff was
recoverable without going to backup tapes.
-during your work it looks like stuff was left as /usr/local/apache2/svn
During recovery its important that noone else have access for stuff like
this, I do mv /usr/local/apache2/svn /usr/local/apache2/svn.sick .
Then do the repair work and put it back. Double check your
owner/group/permission as that can make it look broken aftewards if root
owns stuff.
----- Original message -----
From: "Frank Freund" <ff...@memocomp.de>
To: users@subversion.tigris.org
Date: Fri, 02 Jul 2004 00:34:19 +0200
Subject: Severe repository database corruption
Hi all,
today morning, and after my last commit yesterday (and noone accessed
the repository in any way in between) worked fine, I got the following
error message while trying to commit (using TortoiseSVN):
> "DB_RUNRECOVERY: Fatal error, run database recovery."
Checking the repository via the browser, I got: "Could not open the
requested SVN filesystem", so I started reading docs, mailing lists and
searched the web.
To anticipate, none of the things I tried worked :-( and it seems, that
the database is the problem (but maybe not the reason)
First I will give some information about the environment, then the
things I tried and finally some of the sources I found concerning (other
people having) this problem.
Environment
-------------------
The repository was running now for about half a year with nearly 400
commits and 700MB repository size.
The System configuration is:
SuSE 8.2 with
* Subversion 0.35.1 (now upgraded to 1.0.5, but the problem remains
exactly the same)
* BDB 4.2.52.1
* Apache 2.0.48
All above build from source, and the clientside
* Tortoise SVN 0.24.0 build 554
Attempts
-------------------
First I tried
> linux: # svnadmin recover /usr/local/apache2/svn
> Please wait; recovering the repository may take some time...
> Recovery completed.
> The latest repos revision is 394.
which looked fine, but didn't worked - same errors as above. So next was
> linux: # svnadmin verify /usr/local/apache2/svn
which also completed successfully, but didn't helped either. Next step
were the databasetools:
> linux: # ./db_recover -h /usr/local/apache2/svn/db
This worked, but I just could access the repository over the browser for
some tries, before the same error occurred again. (reproducable always)
Accessing the repository using TorsoiseSVN, I just tried checkouts,
failed directly (reproducable always)
Tried again:
> linux: # svnadmin verify /usr/local/apache2/svn
> svn: Berkeley DB error while opening environment for filesystem
/usr/local/apache2/svn/db:
> DB_RUNRECOVERY: Fatal error, run database recovery
and
> linux: # ./db_recover -h /usr/local/apache2/svn/db
which worked again for short (sometimes 10 seconds, never longer than
about 2 minutes). Some tries later the verify failed:
> linux: # svnadmin verify /usr/local/apache2/svn
> Verified revision 0.
> svn: Berkeley DB error while beginning Berkeley DB transaction for
filesystem /usr/local/apache2/svn/db:
> DB_RUNRECOVERY: Fatal error, run database recovery
> svn: Berkeley DB error while closing 'nodes' database for filesystem
/usr/local/apache2/svn/memocomp/db:
> DB_RUNRECOVERY: Fatal error, run database recovery
Meanwhile updated to SVN1.0.5 and imported the backup (Jesus saves:-)
successfully, but the symptoms remain the same.
Some posts mentioned problems with mixed up libdb's, but as far as I
know a
> linux: # grep libdb /proc/17455/maps
> 4002a000-400e3000 r-xp 00000000 08:02 42429
/usr/local/BerkeleyDB.4.2/lib/libdb-4.2.so
> 400e3000-400e5000 rw-p 000b8000 08:02 42429
/usr/local/BerkeleyDB.4.2/lib/libdb-4.2.so
with 17455 beeing a httpd-PID, confirms me that this isn't the case.
The http-error log shows:
> [Thu Jul 01 08:11:02 2004] [error] [client 192.168.100.11] Could not
fetch resource information. [500, #0]
> [Thu Jul 01 08:11:02 2004] [error] [client 192.168.100.11] Could not
determine the proper revision to access [500, #160029]
> [Thu Jul 01 08:11:02 2004] [error] [client 192.168.100.11]
(13)Permission denied: Berkeley DB error while checkpointing after
Berkeley DB transaction for filesystem /usr/local/apache2/svn/db:
> DB_RUNRECOVERY: Fatal error, run database recovery [500, #160029]
> [Thu Jul 01 08:11:02 2004] [error] [client 192.168.100.11]
(20014)Error string not specified yet: Berkeley DB error while closing
'nodes' database for filesystem /usr/local/apache2/svn/db:
> DB_RUNRECOVERY: Fatal error, run database recovery
> [Thu Jul 01 08:17:02 2004] [error] [client 192.168.100.11]
(20014)Error string not specified yet: Berkeley DB error while opening
environment for filesystem /usr/local/apache2/svn/db:
> DB_RUNRECOVERY: Fatal error, run database recovery, referer:
http://192.168.100.2/repository.html
and later
> [Thu Jul 01 20:14:46 2004] [error] [client 192.168.100.11] Could not
fetch resource information. [500, #0], referer:
http://192.168.100.2/repository.html
> [Thu Jul 01 20:14:46 2004] [error] [client 192.168.100.11] Could not
open the requested SVN filesystem [500, #160029], referer:
http://192.168.100.2/repository.html
> [Thu Jul 01 20:14:46 2004] [error] [client 192.168.100.11] (17)File
exists: Could not open the requested SVN filesystem [500, #160029],
referer: http://192.168.100.2/repository.html
> [Thu Jul 01 20:14:47 2004] [error] [client 192.168.100.11]
(20014)Error string not specified yet: Berkeley DB error while opening
environment for filesystem /usr/local/apache2/svn/db:
> DB_RUNRECOVERY: Fatal error, run database recovery, referer:
http://192.168.100.2/repository.html
Further readings
-------------------
Repository recovery:
http://svnbook.red-bean.com/svnbook/ch05s03.html#svn-ch-5-sect-3.4
Another broken BDB:
http://www.contactor.se/~dast/svnusers/archive-2004-05/0556.shtml
Corrupted repository db:
http://www.contactor.se/~dast/svn/archive-2004-05/0071.shtml
Corrupted repository db:
http://www.contactor.se/~dast/svn/archive-2004-05/0097.shtml
Unrecoverable databases:
http://www.contactor.se/~dast/svnusers/archive-2004-05/1283.shtml
http://subversion.tigris.org/servlets/BrowseList?list=users&by=thread&from=202660
So that's all and I like to thank everyone who has read this &%&$*long
post.
Thanks in advance for any help!
Frank
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Severe repository database corruption
Posted by Ben Collins-Sussman <su...@collab.net>.
On Thu, 2004-07-01 at 17:34, Frank Freund wrote:
> > linux: # svnadmin recover /usr/local/apache2/svn
> > Please wait; recovering the repository may take some time...
> > Recovery completed.
> > The latest repos revision is 394.
> which looked fine, but didn't worked - same errors as above.
What do you mean it didn't work? It *did* work, the repository was
recovered.
Did it get 'stuck' again when you tried to access it? If so, it's
probably because you ran recovery as the root user, which messed up all
the permissions.
Have you read this FAQ?
http://subversion.tigris.org/project_faq.html#permissions
If 'svnadmin recover' doesn't work, then use the catastrophic flag to
db_recover. ('db_recover -vec -h repos/db/'). Then make sure all the
permissions are set correctly before restarting apache. That is, make
sure apache has full read/write access to the whole repository (even
better if apache owns the repository) before starting it again. If the
repository has weird ownership or permissions, then the repository will
get 'stuck' again when apache tries to open it.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org