You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Mark Slater <ma...@analogsoftware.com> on 2004/05/02 03:03:23 UTC

Corrupted repository db

I posted this problem to the user's list, but heard only silence... I  
also posted on the berkeley DB newsgroup and got an answer that, while  
the command worked, failed to restore functionality to the repository.  
So here goes...

A few days ago, I went to shut my computer off (MacOS X 10.3.3, Dual  
G4)... after 10-20 minutes of being logged out, but not shut off, I  
decided it needed help, so I pushed the restart button. I'm not sure if  
that has anything to do with the problem though... it had been a while  
since my last checkin for the day. Anyway, the next day, two of my  
subversion repositories were corrupted. One was fixed with svnadmin  
--recover. The other one seems to be much worse off.

Here are some of the error messages I'm getting:

$ svnadmin verify ./RD
* Verified revision 0.
* Verified revision 1.
svn: Checksum mismatch on rep 'u':
    expected:  34ebc892de3302284606f110c3e573ca
      actual:  13796ec7e50d0e0a50048df550db0758

$ sudo svnadmin recover ./RD
Please wait; recovering the repository may take some time...
svn: DB_RUNRECOVERY: Fatal error, run database recovery

$ db_verify nodes
db_verify: Page 2: unreferenced page
db_verify: Page 3: unreferenced page
db_verify: DB->verify: nodes: DB_VERIFY_BAD: Database verification  
failed

$ db_verify representations db_verify: Page 2: unreferenced page
db_verify: Page 3: unreferenced page
db_verify: DB->verify: representations: DB_VERIFY_BAD: Database  
verification failed

$ db_verify strings
db_verify: Page 58: first page in overflow chain has a prev_pgno 57
db_verify: Page 111: overflow item incomplete
db_verify: Page 21: incorrect next_pgno 61 found in leaf chain (should  
be 91)
db_verify: Page 58: overflow page linked twice from leaf or data page
db_verify: Page 38: overflow page linked twice from leaf or data page
db_verify: Page 54: overflow page of invalid type 5
db_verify: Page 55: overflow page linked twice from leaf or data page
db_verify: DB->verify: strings: DB_VERIFY_BAD: Database verification  
failed

$ sudo db_recover
db_recover: Log sequence error: page LSN 2 322820; previous LSN 2 885276
db_recover: Recovery function for LSN 2 887886 failed on forward pass
db_recover: PANIC: Invalid argument
db_recover: PANIC: fatal region error detected; run recovery
db_recover: PANIC: fatal region error detected; run recovery
db_recover: PANIC: fatal region error detected; run recovery
db_recover: PANIC: fatal region error detected; run recovery
db_recover: PANIC: fatal region error detected; run recovery
db_recover: PANIC: fatal region error detected; run recovery
db_recover: PANIC: fatal region error detected; run recovery
db_recover: PANIC: fatal region error detected; run recovery
db_recover: PANIC: fatal region error detected; run recovery
db_recover: PANIC: fatal region error detected; run recovery
db_recover: PANIC: fatal region error detected; run recovery
db_recover: PANIC: fatal region error detected; run recovery
db_recover: PANIC: fatal region error detected; run recovery
db_recover: PANIC: fatal region error detected; run recovery
db_recover: PANIC: fatal region error detected; run recovery
db_recover: PANIC: fatal region error detected; run recovery
db_recover: PANIC: fatal region error detected; run recovery
db_recover: PANIC: fatal region error detected; run recovery
db_recover: PANIC: fatal region error detected; run recovery
db_recover: PANIC: fatal region error detected; run recovery
db_recover: PANIC: fatal region error detected; run recovery
db_recover: PANIC: fatal region error detected; run recovery
db_recover: PANIC: fatal region error detected; run recovery
db_recover: PANIC: fatal region error detected; run recovery
db_recover: DB_ENV->open: DB_RUNRECOVERY: Fatal error, run database  
recovery

Here's the question and answer from the Berkeley DB newsgroup:
http://groups.google.com/groups?dq=&hl=en&lr=&ie=UTF-8&oe=UTF 
-8&threadm=adecb6f.0404291101.4380b1c9%40posting.google.com&prev=/ 
groups%3Fhl%3Den%26lr%3D%26ie%3DUTF-8%26oe%3DUTF 
-8%26group%3Dcomp.databases.berkeley-db

If there's anything else I can provide that will help determine what  
the problem is, how to fix it, and maybe even what caused it, please  
let me know.

Mark


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: Corrupted repository db

Posted by Branko Čibej <br...@xbc.nu>.
Mark Slater wrote:

> I actually don't think Subversion is the culprit... unless for some 
> reason it failed to exit in the last commit and kept the database open 
> without flushing some of the changes. My hope in posting here is that 
> someone has come across a related problem on their own system and 
> knows enough about the database system to point me on the path that 
> gets the repository working again. The database folks know Berkeley 
> DB, but not how Subversion uses it.
>
> To answer your question, the db_dump -r (or -R) do seem to restore the 
> database (at least temporarily), but it fails to restore the 
> repository. The fun part is that if I try to run svnadmin verify at 
> the end of the following command sequence, I get into a cycle; simply 
> trying to verify the repository (with an apparently recovered 
> database) causes the nodes table to become corrupted again. I can 
> repair it, sure, but it doesn't stay that way.
>
> $ svnadmin verify ~/SVN_Documents/DB
> * Verified revision 0.
> * Verified revision 1.
> svn: Checksum mismatch on rep '3':
>    expected:  4286871d7bb225bda5f931bb91cb3ead
>      actual:  b54cbed34fb8519f8fa4bb47c1de48ee
>
> svn: Berkeley DB error while closing 'nodes' database for filesystem 
> /.../SVN_Documents/RD/db:
> DB_RUNRECOVERY: Fatal error, run database recovery

That's strange. It looks like the nodes table is physically scrambled in 
ways that db_recover can't really fix. "svnadmin verify" doesn't change 
anything in the nodes table, for all practical purposes it's the same as 
"svnadmin dump". Is it possible that your disk happens to have grown a 
bad sector somewhere in the nodes file, or that the volume is full?

> $ db_verify nodes
> db_verify: PANIC: fatal region error detected; run recovery
>
> $ sudo db_recover -v
> Password:
> db_recover: Finding last valid log LSN: file: 3 offset 401252
> db_recover: Recovery starting from [3][401128]
> db_recover: Recovery complete at Sun May  2 02:42:15 2004
> db_recover: Maximum transaction ID 80000008 Recovery checkpoint 
> [3][401252]
>
> $ db_verify nodes
>
> I've looked at the output of the db_dump command, and there's no way I 
> could edit it.... it's lots and lots of numbers that simply don't mean 
> anything to me. Is there any way to know which file is causing the 
> specific error (checksum mismatch) from svnadmin verify?

Try db_dump with the -p option; you'll see the contents in printable 
format, and you can easily interpret the skels. the last string in the 
file or dir skels is the "rep key"; find the one that matches the number 
in the checksum mismatch error, and you've found your file. Also look at 
http://svn.collab.net/repos/svn/trunk/subversion/libsvn_fs_base/notes/structure 
which explains the skel format and will help you to understand what 
you're seeing.

Unfortunately, if the scrambled representation is a delta source, it 
will affect other versions of that file.

-- Brane



---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: Corrupted repository db

Posted by Mark Slater <ma...@analogsoftware.com>.
I actually don't think Subversion is the culprit... unless for some 
reason it failed to exit in the last commit and kept the database open 
without flushing some of the changes. My hope in posting here is that 
someone has come across a related problem on their own system and knows 
enough about the database system to point me on the path that gets the 
repository working again. The database folks know Berkeley DB, but not 
how Subversion uses it.

To answer your question, the db_dump -r (or -R) do seem to restore the 
database (at least temporarily), but it fails to restore the 
repository. The fun part is that if I try to run svnadmin verify at the 
end of the following command sequence, I get into a cycle; simply 
trying to verify the repository (with an apparently recovered database) 
causes the nodes table to become corrupted again. I can repair it, 
sure, but it doesn't stay that way.

$ svnadmin verify ~/SVN_Documents/DB
* Verified revision 0.
* Verified revision 1.
svn: Checksum mismatch on rep '3':
    expected:  4286871d7bb225bda5f931bb91cb3ead
      actual:  b54cbed34fb8519f8fa4bb47c1de48ee

svn: Berkeley DB error while closing 'nodes' database for filesystem 
/.../SVN_Documents/RD/db:
DB_RUNRECOVERY: Fatal error, run database recovery

$ db_verify nodes
db_verify: PANIC: fatal region error detected; run recovery

$ sudo db_recover -v
Password:
db_recover: Finding last valid log LSN: file: 3 offset 401252
db_recover: Recovery starting from [3][401128]
db_recover: Recovery complete at Sun May  2 02:42:15 2004
db_recover: Maximum transaction ID 80000008 Recovery checkpoint 
[3][401252]

$ db_verify nodes

I've looked at the output of the db_dump command, and there's no way I 
could edit it.... it's lots and lots of numbers that simply don't mean 
anything to me. Is there any way to know which file is causing the 
specific error (checksum mismatch) from svnadmin verify?

I'd be happy if I could even get some of the revisions out.

Mark

On May 2, 2004, at 1:34 AM, Branko Čibej wrote:

> Mark Slater wrote:
>
>> I posted this problem to the user's list, but heard only silence... I 
>>  also posted on the berkeley DB newsgroup and got an answer that, 
>> while  the command worked, failed to restore functionality to the 
>> repository.  So here goes...
>>
>> A few days ago, I went to shut my computer off (MacOS X 10.3.3, Dual  
>> G4)... after 10-20 minutes of being logged out, but not shut off, I  
>> decided it needed help, so I pushed the restart button. I'm not sure 
>> if  that has anything to do with the problem though... it had been a 
>> while  since my last checkin for the day. Anyway, the next day, two 
>> of my  subversion repositories were corrupted. One was fixed with 
>> svnadmin  --recover. The other one seems to be much worse off.
>
> I don't see how Subversion can be the culprit here? This looks like an 
> OS bug to me, not flushing file caches to disk or something.
>
>> Here's the question and answer from the Berkeley DB newsgroup:
>> http://groups.google.com/groups?dq=&hl=en&lr=&ie=UTF-8&oe=UTF 
>> -8&threadm=adecb6f.0404291101.4380b1c9%40posting.google.com&prev=/ 
>> groups%3Fhl%3Den%26lr%3D%26ie%3DUTF-8%26oe%3DUTF 
>> -8%26group%3Dcomp.databases.berkeley-db
>
> So, have you tried db_dump -r?
>
>> If there's anything else I can provide that will help determine what  
>> the problem is, how to fix it, and maybe even what caused it, please  
>> let me know.
>
> I don't think anyone here can help you. It doesn't look like the data 
> corruption was caused by Subversion, but by the combination of you OS 
> no flushing buffers and you turning off the power. I suggest you do 
> what the BDB people proposed; db_dump -r, and if that doesn't help, 
> then db_dump -R and edit the result -- although I'm not sure how...
>
> -- Brane
>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org


Re: Corrupted repository db

Posted by Branko Čibej <br...@xbc.nu>.
Mark Slater wrote:

> I posted this problem to the user's list, but heard only silence... I  
> also posted on the berkeley DB newsgroup and got an answer that, 
> while  the command worked, failed to restore functionality to the 
> repository.  So here goes...
>
> A few days ago, I went to shut my computer off (MacOS X 10.3.3, Dual  
> G4)... after 10-20 minutes of being logged out, but not shut off, I  
> decided it needed help, so I pushed the restart button. I'm not sure 
> if  that has anything to do with the problem though... it had been a 
> while  since my last checkin for the day. Anyway, the next day, two of 
> my  subversion repositories were corrupted. One was fixed with 
> svnadmin  --recover. The other one seems to be much worse off.

I don't see how Subversion can be the culprit here? This looks like an 
OS bug to me, not flushing file caches to disk or something.

> Here's the question and answer from the Berkeley DB newsgroup:
> http://groups.google.com/groups?dq=&hl=en&lr=&ie=UTF-8&oe=UTF 
> -8&threadm=adecb6f.0404291101.4380b1c9%40posting.google.com&prev=/ 
> groups%3Fhl%3Den%26lr%3D%26ie%3DUTF-8%26oe%3DUTF 
> -8%26group%3Dcomp.databases.berkeley-db

So, have you tried db_dump -r?

> If there's anything else I can provide that will help determine what  
> the problem is, how to fix it, and maybe even what caused it, please  
> let me know.

I don't think anyone here can help you. It doesn't look like the data 
corruption was caused by Subversion, but by the combination of you OS no 
flushing buffers and you turning off the power. I suggest you do what 
the BDB people proposed; db_dump -r, and if that doesn't help, then 
db_dump -R and edit the result -- although I'm not sure how...

-- Brane



---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org