You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Mark Slater <ma...@analogsoftware.com> on 2004/05/02 03:03:23 UTC
Corrupted repository db
I posted this problem to the user's list, but heard only silence... I
also posted on the berkeley DB newsgroup and got an answer that, while
the command worked, failed to restore functionality to the repository.
So here goes...
A few days ago, I went to shut my computer off (MacOS X 10.3.3, Dual
G4)... after 10-20 minutes of being logged out, but not shut off, I
decided it needed help, so I pushed the restart button. I'm not sure if
that has anything to do with the problem though... it had been a while
since my last checkin for the day. Anyway, the next day, two of my
subversion repositories were corrupted. One was fixed with svnadmin
--recover. The other one seems to be much worse off.
Here are some of the error messages I'm getting:
$ svnadmin verify ./RD
* Verified revision 0.
* Verified revision 1.
svn: Checksum mismatch on rep 'u':
expected: 34ebc892de3302284606f110c3e573ca
actual: 13796ec7e50d0e0a50048df550db0758
$ sudo svnadmin recover ./RD
Please wait; recovering the repository may take some time...
svn: DB_RUNRECOVERY: Fatal error, run database recovery
$ db_verify nodes
db_verify: Page 2: unreferenced page
db_verify: Page 3: unreferenced page
db_verify: DB->verify: nodes: DB_VERIFY_BAD: Database verification
failed
$ db_verify representations db_verify: Page 2: unreferenced page
db_verify: Page 3: unreferenced page
db_verify: DB->verify: representations: DB_VERIFY_BAD: Database
verification failed
$ db_verify strings
db_verify: Page 58: first page in overflow chain has a prev_pgno 57
db_verify: Page 111: overflow item incomplete
db_verify: Page 21: incorrect next_pgno 61 found in leaf chain (should
be 91)
db_verify: Page 58: overflow page linked twice from leaf or data page
db_verify: Page 38: overflow page linked twice from leaf or data page
db_verify: Page 54: overflow page of invalid type 5
db_verify: Page 55: overflow page linked twice from leaf or data page
db_verify: DB->verify: strings: DB_VERIFY_BAD: Database verification
failed
$ sudo db_recover
db_recover: Log sequence error: page LSN 2 322820; previous LSN 2 885276
db_recover: Recovery function for LSN 2 887886 failed on forward pass
db_recover: PANIC: Invalid argument
db_recover: PANIC: fatal region error detected; run recovery
db_recover: PANIC: fatal region error detected; run recovery
db_recover: PANIC: fatal region error detected; run recovery
db_recover: PANIC: fatal region error detected; run recovery
db_recover: PANIC: fatal region error detected; run recovery
db_recover: PANIC: fatal region error detected; run recovery
db_recover: PANIC: fatal region error detected; run recovery
db_recover: PANIC: fatal region error detected; run recovery
db_recover: PANIC: fatal region error detected; run recovery
db_recover: PANIC: fatal region error detected; run recovery
db_recover: PANIC: fatal region error detected; run recovery
db_recover: PANIC: fatal region error detected; run recovery
db_recover: PANIC: fatal region error detected; run recovery
db_recover: PANIC: fatal region error detected; run recovery
db_recover: PANIC: fatal region error detected; run recovery
db_recover: PANIC: fatal region error detected; run recovery
db_recover: PANIC: fatal region error detected; run recovery
db_recover: PANIC: fatal region error detected; run recovery
db_recover: PANIC: fatal region error detected; run recovery
db_recover: PANIC: fatal region error detected; run recovery
db_recover: PANIC: fatal region error detected; run recovery
db_recover: PANIC: fatal region error detected; run recovery
db_recover: PANIC: fatal region error detected; run recovery
db_recover: PANIC: fatal region error detected; run recovery
db_recover: DB_ENV->open: DB_RUNRECOVERY: Fatal error, run database
recovery
Here's the question and answer from the Berkeley DB newsgroup:
http://groups.google.com/groups?dq=&hl=en&lr=&ie=UTF-8&oe=UTF
-8&threadm=adecb6f.0404291101.4380b1c9%40posting.google.com&prev=/
groups%3Fhl%3Den%26lr%3D%26ie%3DUTF-8%26oe%3DUTF
-8%26group%3Dcomp.databases.berkeley-db
If there's anything else I can provide that will help determine what
the problem is, how to fix it, and maybe even what caused it, please
let me know.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Corrupted repository db
Posted by Branko Čibej <br...@xbc.nu>.
Mark Slater wrote:
> I actually don't think Subversion is the culprit... unless for some
> reason it failed to exit in the last commit and kept the database open
> without flushing some of the changes. My hope in posting here is that
> someone has come across a related problem on their own system and
> knows enough about the database system to point me on the path that
> gets the repository working again. The database folks know Berkeley
> DB, but not how Subversion uses it.
>
> To answer your question, the db_dump -r (or -R) do seem to restore the
> database (at least temporarily), but it fails to restore the
> repository. The fun part is that if I try to run svnadmin verify at
> the end of the following command sequence, I get into a cycle; simply
> trying to verify the repository (with an apparently recovered
> database) causes the nodes table to become corrupted again. I can
> repair it, sure, but it doesn't stay that way.
>
> $ svnadmin verify ~/SVN_Documents/DB
> * Verified revision 0.
> * Verified revision 1.
> svn: Checksum mismatch on rep '3':
> expected: 4286871d7bb225bda5f931bb91cb3ead
> actual: b54cbed34fb8519f8fa4bb47c1de48ee
>
> svn: Berkeley DB error while closing 'nodes' database for filesystem
> /.../SVN_Documents/RD/db:
> DB_RUNRECOVERY: Fatal error, run database recovery
That's strange. It looks like the nodes table is physically scrambled in
ways that db_recover can't really fix. "svnadmin verify" doesn't change
anything in the nodes table, for all practical purposes it's the same as
"svnadmin dump". Is it possible that your disk happens to have grown a
bad sector somewhere in the nodes file, or that the volume is full?
> $ db_verify nodes
> db_verify: PANIC: fatal region error detected; run recovery
>
> $ sudo db_recover -v
> Password:
> db_recover: Finding last valid log LSN: file: 3 offset 401252
> db_recover: Recovery starting from [3][401128]
> db_recover: Recovery complete at Sun May 2 02:42:15 2004
> db_recover: Maximum transaction ID 80000008 Recovery checkpoint
> [3][401252]
>
> $ db_verify nodes
>
> I've looked at the output of the db_dump command, and there's no way I
> could edit it.... it's lots and lots of numbers that simply don't mean
> anything to me. Is there any way to know which file is causing the
> specific error (checksum mismatch) from svnadmin verify?
Try db_dump with the -p option; you'll see the contents in printable
format, and you can easily interpret the skels. the last string in the
file or dir skels is the "rep key"; find the one that matches the number
in the checksum mismatch error, and you've found your file. Also look at
http://svn.collab.net/repos/svn/trunk/subversion/libsvn_fs_base/notes/structure
which explains the skel format and will help you to understand what
you're seeing.
Unfortunately, if the scrambled representation is a delta source, it
will affect other versions of that file.
-- Brane
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Corrupted repository db
Posted by Mark Slater <ma...@analogsoftware.com>.
I actually don't think Subversion is the culprit... unless for some
reason it failed to exit in the last commit and kept the database open
without flushing some of the changes. My hope in posting here is that
someone has come across a related problem on their own system and knows
enough about the database system to point me on the path that gets the
repository working again. The database folks know Berkeley DB, but not
how Subversion uses it.
To answer your question, the db_dump -r (or -R) do seem to restore the
database (at least temporarily), but it fails to restore the
repository. The fun part is that if I try to run svnadmin verify at the
end of the following command sequence, I get into a cycle; simply
trying to verify the repository (with an apparently recovered database)
causes the nodes table to become corrupted again. I can repair it,
sure, but it doesn't stay that way.
$ svnadmin verify ~/SVN_Documents/DB
* Verified revision 0.
* Verified revision 1.
svn: Checksum mismatch on rep '3':
expected: 4286871d7bb225bda5f931bb91cb3ead
actual: b54cbed34fb8519f8fa4bb47c1de48ee
svn: Berkeley DB error while closing 'nodes' database for filesystem
/.../SVN_Documents/RD/db:
DB_RUNRECOVERY: Fatal error, run database recovery
$ db_verify nodes
db_verify: PANIC: fatal region error detected; run recovery
$ sudo db_recover -v
Password:
db_recover: Finding last valid log LSN: file: 3 offset 401252
db_recover: Recovery starting from [3][401128]
db_recover: Recovery complete at Sun May 2 02:42:15 2004
db_recover: Maximum transaction ID 80000008 Recovery checkpoint
[3][401252]
$ db_verify nodes
I've looked at the output of the db_dump command, and there's no way I
could edit it.... it's lots and lots of numbers that simply don't mean
anything to me. Is there any way to know which file is causing the
specific error (checksum mismatch) from svnadmin verify?
I'd be happy if I could even get some of the revisions out.
Mark
On May 2, 2004, at 1:34 AM, Branko Čibej wrote:
> Mark Slater wrote:
>
>> I posted this problem to the user's list, but heard only silence... I
>> also posted on the berkeley DB newsgroup and got an answer that,
>> while the command worked, failed to restore functionality to the
>> repository. So here goes...
>>
>> A few days ago, I went to shut my computer off (MacOS X 10.3.3, Dual
>> G4)... after 10-20 minutes of being logged out, but not shut off, I
>> decided it needed help, so I pushed the restart button. I'm not sure
>> if that has anything to do with the problem though... it had been a
>> while since my last checkin for the day. Anyway, the next day, two
>> of my subversion repositories were corrupted. One was fixed with
>> svnadmin --recover. The other one seems to be much worse off.
>
> I don't see how Subversion can be the culprit here? This looks like an
> OS bug to me, not flushing file caches to disk or something.
>
>> Here's the question and answer from the Berkeley DB newsgroup:
>> http://groups.google.com/groups?dq=&hl=en&lr=&ie=UTF-8&oe=UTF
>> -8&threadm=adecb6f.0404291101.4380b1c9%40posting.google.com&prev=/
>> groups%3Fhl%3Den%26lr%3D%26ie%3DUTF-8%26oe%3DUTF
>> -8%26group%3Dcomp.databases.berkeley-db
>
> So, have you tried db_dump -r?
>
>> If there's anything else I can provide that will help determine what
>> the problem is, how to fix it, and maybe even what caused it, please
>> let me know.
>
> I don't think anyone here can help you. It doesn't look like the data
> corruption was caused by Subversion, but by the combination of you OS
> no flushing buffers and you turning off the power. I suggest you do
> what the BDB people proposed; db_dump -r, and if that doesn't help,
> then db_dump -R and edit the result -- although I'm not sure how...
>
> -- Brane
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Corrupted repository db
Posted by Branko Čibej <br...@xbc.nu>.
Mark Slater wrote:
> I posted this problem to the user's list, but heard only silence... I
> also posted on the berkeley DB newsgroup and got an answer that,
> while the command worked, failed to restore functionality to the
> repository. So here goes...
>
> A few days ago, I went to shut my computer off (MacOS X 10.3.3, Dual
> G4)... after 10-20 minutes of being logged out, but not shut off, I
> decided it needed help, so I pushed the restart button. I'm not sure
> if that has anything to do with the problem though... it had been a
> while since my last checkin for the day. Anyway, the next day, two of
> my subversion repositories were corrupted. One was fixed with
> svnadmin --recover. The other one seems to be much worse off.
I don't see how Subversion can be the culprit here? This looks like an
OS bug to me, not flushing file caches to disk or something.
> Here's the question and answer from the Berkeley DB newsgroup:
> http://groups.google.com/groups?dq=&hl=en&lr=&ie=UTF-8&oe=UTF
> -8&threadm=adecb6f.0404291101.4380b1c9%40posting.google.com&prev=/
> groups%3Fhl%3Den%26lr%3D%26ie%3DUTF-8%26oe%3DUTF
> -8%26group%3Dcomp.databases.berkeley-db
So, have you tried db_dump -r?
> If there's anything else I can provide that will help determine what
> the problem is, how to fix it, and maybe even what caused it, please
> let me know.
I don't think anyone here can help you. It doesn't look like the data
corruption was caused by Subversion, but by the combination of you OS no
flushing buffers and you turning off the power. I suggest you do what
the BDB people proposed; db_dump -r, and if that doesn't help, then
db_dump -R and edit the result -- although I'm not sure how...
-- Brane
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org